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ABSTRACT 


The  Central  Test  and  Evaluation  Investment  Program  (CTEIP)  was  established  in 
1990  by  the  Office  of  the  Secretary  of  Defense  (OSD),  in  response  to  Congressional 
direction,  to  provide  a  corporate  investment  approach  to  meeting  Service  and  Defense 
Agency  Test  and  Evaluation  (T&E)  needs.  This  approach  to  the  allocation  of  test 
resources  increases  interoperability  between  the  Services,  and  interconnectivity  among 
the  test  centers  and  ranges.  It  serves  to  focus  T&E  expertise  on  test  requirements  that  are 
of  the  highest  priority.  The  CTEIP  project  selection  process  consists  of  a  cyclic  approach 
in  which  the  Services  and  Defense  Agencies  solicit  T&E  needs,  propose  solutions  to 
those  needs,  and  then  formally  project  proposals.  Program  Management  Offices  (PMOs) 
are  not  directly  solicited  for  potential  project  solutions  addressing  direct  operational  test 
needs,  yet,  the  PMOs  are  facing  significantly  reduced  T&E  funding.  Knowledge  of  the 
CTEIP  program  and  the  proposal  submission  process  should  aid  the  PMOs  in  the 
planning  and  execution  of  their  Operational  Test  Programs.  This  thesis  reviews  the 
CTEIP  and  evaluates  the  extent  to  which  PMOs  are  involved  in  the  submission  of  T&E 
needs,  and  the  value  of  PMO  involvement  in  CTEIP  project  selection  process. 
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I.  INTRODUCTION 


A.  PURPOSE 

This  thesis  reviews  the  Central  Test  and  Evaluation  Investment  Program  (CTEIP) 
and  evaluates  the  extent  to  which  Program  Management  Offices  (PMO)  are  involved  in 
the  submission  of  test  and  evaluation  needs,  from  which  CTEIP  projects  are  selected. 
The  objective  is  to  determine  if  the  CTEIP  managers  for  potential  projects  directly  or 
indirectly  solicit  PMOs,  and  how  the  PMOs  can  more  directly  insert  themselves  into  the 
project  selection  process.  Data  is  gathered  to  detennine  if  there  is  value  added  for  the 
PMOs  to  participate  in  CTEIP.  The  research  includes  conducting  a  detailed  review  of  the 
CTEIP  selection  process  and  participating  organizations,  and  conducting  a  series  of 
telephone  and  personal  interviews  with  PMO  Test  Division  Chiefs  and  Team  Leads  as  to 
their  knowledge  of  CTEIP  and  solicitations  they  have  participated  in  regarding  Test  and 
Evaluation  (T&E)  needs. 

B.  BACKGROUND 

The  CTEIP  was  established  in  1990  by  the  Office  of  the  Secretary  of  Defense 
(OSD),  in  response  to  Congressional  direction,  to  provide  a  corporate  investment 
approach  to  meeting  Service  and  Defense  Agency  T&E  needs.  This  approach  to  the 
allocation  of  test  resources  increases  interoperability  between  the  Services,  and 
interconnectivity  among  the  test  centers  and  ranges.  It  serves  to  focus  T&E  expertise  on 
test  requirements  that  are  of  the  highest  priority.  The  CTEIP  charter  also  promotes  joint 
initiatives  and  elimination  of  unwarranted  duplication  of  effort.  Three  foundation  criteria 
for  CTEIP  projects  were  established  by  Congressional  language:  must  have  multi-service 
applicability;  must  be  developmental  in  nature;  and  must  not  be  used  for  procurement. 

Oversight  of  the  CTEIP  projects  resides  within  the  Office  of  the  Director, 
Operational  Test  and  Evaluation  (DOT&E).  The  CTEIP  project  selection  process 
consists  of  a  cyclic  approach  in  which  the  Services  and  Defense  Agencies  solicit  T&E 
needs,  propose  solutions  to  those  needs,  and  then  formally  project  proposals.  Each  stage 
of  the  process  requires  review,  analysis,  and  prioritization  of  the  proposals  in  order  to 
determine  the  areas  of  overall  T&E  priorities  within  the  Services  and  defense  agencies. 
Typically,  solicitations  for  these  proposals  are  addressed  within  the  Service  and  Defense 
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Agency  T&E  organizations,  with  the  only  input  from  PMOs  coming  from  comments 
included  in  Test  and  Evaluation  Management  Plans  (TEMPs).  The  CTEIP  management 
does  not  directly  solicit  PMOs  for  potential  project  solutions  addressing  direct 
operational  test  needs. 

However,  PMOs  are  facing  significantly  reduced  T&E  funding.  Knowledge  of 
the  CTEIP  program  and  the  proposal  submission  process  can  aid  the  PMOs  in  the 
planning  and  execution  of  their  Operational  Test  Programs.  An  analysis  of  the  total 
CTEIP  program  is  conducted  within  this  research  to  establish  the  most  efficient  means  of 
inserting  PMO’s  proposals  into  the  CTEIP  project  planning  process. 

C.  THESIS  OBJECTIVES 

The  benefits  of  this  study  are  intended  to  be  two-fold.  First,  a  review  of  the 
CTEIP  project  selection  process  to  determine  the  extent  to  which  PMOs  are  involved  will 
be  helpful  to  the  CTEIP  managers  for  future  planning.  Secondly,  a  review  of  the  CTEIP 
project  selection  process  will  help  clarify  the  process  for  PMO  test  leads  and  will  better 
define  the  path  a  PMO  needs  to  take  to  submit  a  proposal  for  acceptance  as  a  CTEIP 
funded  program.  This  process  is  currently  not  clear  for  PMO  test  leads  to  follow 
successfully  without  prior  experience  within  the  CTEIP  process.  Guidance  for  PMOs 
will  be  developed  to  aid  them  in  their  CTEIP  funding  quests. 

D.  RESEARCH  QUESTIONS 

1.  Primary  Research  Questions 

•  How  can  a  Program  Management  Office  (PMO)  become  aware  of  the 
availability  and  applicability  of  Central  Test  and  Evaluation  Investment 
Program  (CTEIP)  funding  for  their  test  programs? 

•  Would  funding  from  CTEIP  be  of  value  to  the  PMO’s  test  program? 

•  How  can  a  Program  Management  Office  insert  themselves  into  the  CTEIP 
needs  and  solution  process  in  order  to  obtain  funding  for  test  program 
developments? 

2.  Secondary  Research  Question 

•  What  are  some  potential  strategies  that  the  CTEIP  managers  could  use  to 
educate  and  offer  assistance  to  PMO  personnel? 

E.  SCOPE  AND  LIMITATIONS 

This  thesis  examines  the  challenges  faced  by  PMO  Test  Leads  as  they  prepare  and 
submit  project  proposals  for  CTEIP  funding  consideration.  U.  S.  Army  PMO  Test  Leads, 
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primarily  from  the  U.S.  Army  Aviation  and  Missile  Command  and  associated  Program 
Executive  Office  PMOs,  are  the  focus  group  for  telephone  interviews  concerning  their 
knowledge  of  and  participation  in  CTEIP  activities  to  date.  Discussions  with  other 
service  PMO  Test  Leads  will  be  limited.  The  CTEIP  management  organization  within 
OSD  will  be  contacted.  The  infonnation  gained  from  interviews  with  OSD  personnel  and 
the  infonnation  gained  from  CTEIP  documentation  will  serve  as  the  baseline  for  the 
current  CTEIP  proposal  submission  process  and  selection  criteria. 

F.  METHODOLOGY 

1.  Data  Collection  Methodology 

The  research  for  this  thesis  was  accomplished  in  three  ways.  Initially,  a 
comprehensive  literature  search  was  conducted  in  which  information  pertaining  to  CTEIP 
policy,  execution  guides,  reports,  briefings,  and  Congressional  language  was  reviewed 
and  analyzed.  Inherent  in  the  documentation  procured  through  the  literature  search  was 
information  on  the  roles  and  responsibilities  of  the  CTEIP  management,  the  CTEIP 
project  selection  process,  and  a  review  of  CTEIP  projects  over  the  last  five  years  as  to 
their  origin  and  sponsorship.  Next,  interviews  were  conducted  with  PMO  Test  Division 
Chiefs  and  T&E  Team  Leads  to  determine  the  extent  to  which  they  had  knowledge  of 
CTEIP,  had  participated  in  CTEIP  project  selection  processes,  or  had  communicated  with 
other  organizations  managing  a  CTEIP  project.  These  PMO  personnel  were  also 
questioned  as  to  what  value  they  would  gain  with  CTEIP  funding  for  portions  of  their  test 
program.  Finally,  CTEIP  management  officials  were  interviewed  to  detennine  the  extent 
to  which  they  actively  solicit  test  needs  directly  from  PMOs,  and  the  value  they  would 
place  on  a  more  interactive  participation  by  PMOs. 

G.  THESIS  ORGANIZATION 

The  thesis  is  organized  into  six  chapters. 

Chapter  I:  Introduction.  This  chapter  presents  the  purpose,  background, 
objective,  research  questions,  scope  and  limitations,  methodology,  and  organization  of 
the  thesis. 

Chapter  II:  CTEIP  Overview.  In  this  chapter,  an  overview  of  the  CTEIP  is 
presented  to  include  its  origin,  purpose,  and  program  objectives.  The  CTEIP 
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management  organization  and  program  structure  and  planning  process  is  also  discussed. 
Finally,  other  Anny  organizations  that  participate  in  the  CTEIP  process  are  presented. 

Chapter  III:  Data  Presentation.  This  chapter  presents  a  review  of  the  Test 
Investment  Process  for  the  total  test  community,  CTEIP,  and  each  of  the  CTEIP  projects. 
Interviews  were  conducted  with  PMO  Test  Division  Chiefs  and  Team  Leads  as  to  their 
knowledge  of  CTEIP.  Finally,  CTEIP  management  organization  representatives  were 
interviewed  as  to  their  source  of  CTEIP  proposals  and  their  solicitation  of  PMOs  for 
CTEIP  projects. 

Chapter  IV:  Data  Analysis.  Analysis  was  conducted  on  the  data  collected  in 
Chapter  III.  This  analysis  provides  insight  into  the  CTEIP  project  selection  process;  the 
value  of  CTEIP  to  PMO,  and  their  knowledge  of  its  existence;  and  the  manner  in  which 
the  CTEIP  management  focuses  their  project  solicitation  process. 

Chapter  VI:  Conclusions  and  Recommendations.  This  chapter  presents 
conclusions  and  recommendations  for  both  PMOs  and  the  CTEIP  management.  The 
answers  to  the  research  questions  are  given.  Also  found  in  this  chapter  are  several  ideas 
for  further  areas  of  study  relating  to  this  thesis. 


4 


II.  CTEIP  PROGRAM  OVERVIEW 


A.  PROGRAM  BEGINNINGS 

Prior  to  1989,  the  Office  of  the  Secretary  of  Defense  (OSD)  was  seeking  a  way  to 
enhance  their  oversight  and  management  of  the  Department  of  Defense  (DoD)  Test  and 
Evaluation  (T&E)  capability  base.  This  quest  was  a  result  of  Congressional  direction  to 
improve  the  way  investments  were  planned  and  coordinated  in  T&E  facilities.  The 
Central  Test  and  Evaluation  Investment  Program  (CTEIP)  was  established  in  1990.  It 
was  given  the  Program  Element  0604940D.  [Ref.  l:p.  1] 

B.  PROGRAM  BACKGROUND 

The  CTEIP  was  established  and  first  funded  in  fiscal  year  (FY)  1991  under  the 
sponsorship  and  oversight  of  the  Deputy  Director,  Defense  Research  and  Engineering 
(Test  and  Evaluation)  (DDDR&E  (T&E)).  Reorganization  in  1992  changed  the 
DDDR&E  (T&E)  to  the  Director,  Test  and  Evaluation  (DT&E)  in  the  Office  of  the  Under 
Secretary  of  Defense  (Acquisition)  (OUSD  (A)).  An  organizational  change  occurred 
again  in  1994  to  establish  the  Office  of  the  Director,  Test  Systems  Engineering  and 
Evaluation  (DTSE&E),  in  the  Office  of  the  Under  Secretary  of  Defense  (Acquisition  and 
Technology)  (OUSD  (A&T)).  In  1999,  another  realignment  took  place  that  transferred 
T&E  functions  from  OUSD  (A&T)  to  the  Director,  Operational  Test  and  Evaluation 
(DOT&E).  [Ref.  2:p.  1] 

The  CTEIP  has  a  funding  level  of  approximately  $125  million  per  year.  [Ref.  3:p. 
3].  This  funding  is  divided  among  three  separately  managed  projects.  The  subprojects  of 
the  Joint  Improvement  and  Modernization  (JIM)  project  comprise  approximately 
seventy-five  per  cent  of  the  total  yearly  funding.  The  Test  Technology  Development  and 
Demonstration  (TTD&D)  subprojects  make  up  approximately  five  per  cent  of  the  total 
yearly  funding.  Each  TTD&D  subproject  is  limited  to  under  $500  thousand  per  year. 
The  Resource  Enhancement  Projects  (REP)  subprojects  are  approximately  twenty  per 
cent  of  the  total  yearly  CTEIP  funding.  [Ref.  4:p.  5]. 

Three  main  criteria  for  CTEIP  projects  were  established  directly  from 
Congressional  language.  To  be  considered  for  CTEIP  funding  a  project  must  have  multi- 
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Service  applicability;  must  be  developmental  in  nature;  and  must  not  be  used  for 
procurement.  [Ref.  l:p.  1] 

C.  CTEIP  PROGRAM  PURPOSE  AND  STRUCTURE 

The  CTEIP  was  established  with  several  main  objectives.  Some  of  these 
objectives  are  applicable  to  all  three  of  the  CTEIP  sub-projects.  Each  of  these  projects, 
JIM,  TTD&D,  and  REP,  has  a  distinct  purpose,  management  structure,  selection  criteria 
and  execution  guidance  within  the  confines  of  the  overall  CTEIP.  This  section  describes 
the  major  features  of  each  of  the  CTEIP  projects.  [Ref.  1  :p.  2] 


Test  Technology 
Development  and 
Demonstration 
(TTD&D) 

Requirements 
1-3  Yrs 

Purpose: 

•  To  investigate  emerging 
technologies  for 
meeting  requirements. 

•  Projects  <  .5M  /  yr 


Joint  Improvement 
&  Modernization 
(JIM) 


Requirements 
3-5  Yrs 

Purpose: 

•  System  development  of  T&E 
capabilities 

•  Must  meet  joint  requirements 

•  Services  must  meet  program 
O&M  and  life  cycle  support 


Resource 

Enhancement  Project 
(REP) 

Requirements 
1-2  Yrs 

Purpose: 

•  Funds  identified  OT  shortfalls 

•  Can  be  program  and  Service 
unique 


75% 


Figure  2.1.  CTEIP  Project  Categories.  [From:  Ref.  3:p.  9] 
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Figure  2.2.  JIM  Management  Structure.  [From:  Ref.  3:p.  10] 

1.  Joint  Improvement  and  Modernization  (JIM)  Project 
a.  Project  Description  and  Objectives 

The  JIM  Project  is  most  concerned  with  the  long-tenn  T&E  infrastructure 
investments.  As  such,  the  sub-projects  in  this  area  must  utilize  the  state-of-the-art  in 
T&E  technologies  to  address  specific  known  deficiencies  in  the  DoD  T&E  capabilities. 
The  improvement  of  interoperability  and  interconnectivity  of  test  assets  across  test 
centers,  ranges,  and  facilities  is  imperative  to  maximize  an  efficient  inter-service  use  of 
the  test  assets.  The  reduction  of  T&E  manpower,  maintenance  requirements  and 
operating  expenses  is  of  prime  importance  in  this  time  of  a  downsized  workforce.  To 
accomplish  this,  ongoing  T&E  technology  development  programs  must  be  established 
and  maintained  to  investigate  areas  of  advanced  technologies  that  could  be  applied  to 
T&E  to  add  technical  capability  and  value.  Targets,  test  instrumentation,  and  threat 
simulators  are  important  parts  of  the  T&E  infrastructure  and  consistency,  commonality, 
and  interoperability  of  these  assets  must  be  achieved  across  the  services.  Modeling  and 
simulation  is  becoming  an  integral  piece  of  T&E  and  must  be  developed,  validated,  and 
integrated  with  open-air  testing  in  order  to  provide  timely,  accurate,  and  cost-effective 
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results.  Mobile  test  instrumentation  should  be  promoted  where  economically  and 
technically  feasible  as  an  alternative  to  fixed  facilities.  Resources  should  be  available  to 
fund  shortfalls  in  critical  near-tenn  operational  test  capabilities.  Past  JIM  projects  have 
addressed  such  issues  as  automated  data  collection,  processing,  display,  and  archiving; 
smart  munitions  testing;  simulation  and  end-game  measurement;  testing  applications  of 
advanced  materials;  test  design;  advanced  sensors;  and  space  systems.  [Ref.  l:p.  2] 

The  JIM  subprojects  comprise  seventy-five  percent  of  the  total  CTEIP 
yearly  funding  [Ref.  4:p.  5].  The  JIM  subprojects  are  usually  multi -phase  in  nature  with 
a  developmental  time  of  approximately  three  to  five  years.  The  subprojects  must  meet 
stringent  criteria  for  joint  requirements.  [Ref.  l:p.  7] 

The  CTEIP  JIM  projects  are  categorized  by  functional  areas  consistent 
with  the  Reliance  areas  used  to  group  test  capabilities  throughout  DoD.  The  areas  are: 
Air  Combat;  Land  Combat;  Sea  Combat;  Space  Combat;  Common  Range 
Instrumentation;  Electronic  Combat;  Armament  and  Munitions;  Targets;  Test 
Environments;  and  Command,  Control,  Communications,  Computers,  and  Intelligence 
(C4I).  [Ref.  l:pp.  3-4] 

b.  Management  Approach 

The  CTEIP  Program  Element  Manager  (PEM)  confirms,  with  preparation 
of  a  Test  Package  Directive  (TPD),  which  service  or  Defense  Agency  will  be  responsible 
for  the  project  execution  of  the  JIM  project.  The  CTEIP  PEM  and  the  OSD  Test 
Investment  Coordinating  Committee  (OTICC)  retain  management  oversight  of  the  JIM 
project  throughout  its  development.  The  day-to-day  project  management  is  left  to  the 
executing  agency  and  an  individually  named  Project  Director  (PD).  Project  progress  is 
monitored  thorough  the  submission  of  monthly  status  reports.  Internal  Program  Reviews 
as  necessary  or  requested  by  the  (OTICC),  and  annual  Mid-Year  Review  briefings. 
Deviations  from  the  approved  Program  Management  Plan  (PMP)  must  have  approval 
from  the  CTEIP  PEM  and  the  OTICC.  [Refs.  1  and  7] 

2.  Test  Technology  Development  &  Demonstration  (TTD&D) 
a.  Project  Description 

TTD&D  projects  facilitate  the  transition  of  technology  as  it  matures  from 

the  laboratories  to  the  T&E  communities.  This  transition  enhances  test  capabilities  and 
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reduces  the  technical  risk  of  testing  future  defense  systems.  Through  these  subprojects, 
equipment  and  methods  are  provided  to  the  T&E  community  to  evaluate  new  weapon 
systems  that  evolve  from  advanced  research  and  development  initiatives.  The  goals  and 
objectives  of  the  projects  are  reviewed  annually  to  ensure  that  the  strategic  areas  of 
interest  are  being  addressed.  [Ref.  5:p.  1]  Yearly  reviews  of  the  projects  ensure  they 
match  the  strategic  thrust  areas.  The  review  also  ensures  that  the  projects  support  DoD 
guidance  and  policy  as  published  in  the  Defense  Technology  Area  Plan.  The  subprojects 
selected  for  TTD&D  projects  are  expected  to  be  available  for  transition  to  fielded 
technology  within  three  years.  The  funding  level  is  limited  to  less  than  $500,000  per 
year.  [Ref.  5:p.  1-2] 

b.  Management  Approach 

The  TTD&D  is  a  centrally  planned  and  funded  project  with  decentralized 
execution.  The  CTEIP  PEM  serves  as  the  Executive  Secretary  of  the  OTICC.  The 
CTEIP  PEM  issues  policy  guidelines  and  direction;  approves  the  yearly  funding;  issues 
the  TPD  for  project  execution;  and  designates  a  TTD&D  PD  to  oversee  the  day-to-day 
activities.  All  this  is  under  the  continual  oversight  of  the  OTICC  and  DOT&E.  [Ref.  5:p. 
2-1]  The  TTD&D  PD  manages  the  day-to-day  operations  of  the  project;  develops  the 
PMP  for  submission  and  approval  by  the  CTEIP  PEM;  coordinates  any  necessary 
subproject  nomination,  evaluation  and  selection  process;  develops  and  manages  to  the 
funding  profile  and  spend  plan;  prepares  an  overall  Annual  Assessment  Report;  conducts 
reviews;  and  prepares  a  yearly  project  status  briefing  at  the  CTEIP  Mid-Year  Review. 
[Ref.  5:p.  2-2] 
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Figure  2.3.  TTD&D  Management  Structure.  [From:  Ref.  3:p.  12] 

3.  Resource  Enhancement  Project  (REP) 
a.  Project  Description 

REP  has  the  main  objective  of  ensuring  that  acquisition  programs  in  the 
services  and  DoD  agencies  have  the  resources  necessary  to  test  in  the  most  realistic 
environment  available.  [Ref.  2:p.  1-1]  The  main  focus  of  the  REP  funds  is  to  answer 
operational  test  requirements  that  will  be  needed  within  three  years  or  less,  that  are  high 
priority,  and  that  were  not  programmed  by  the  program  office  or  by  POM  submissions. 
These  near  term  requirements  are  usually  a  result  of  changes  in  the  test  requirements, 
scope  of  testing,  or  other  program  changes  that  affect  testing.  The  development  and 
evaluation  of  the  weapon  systems  and  system  upgrades  become  high  risk  without  these 
additional  Operational  Test  (OT)  resources.  The  REP  provides  a  more  rapid  response  to 
unforeseen  issues  and  unplanned  test  needs  than  the  other  CTEIP  projects.  [Ref.  2:p.  1-1] 
However,  the  shorter  development  time  is  more  limiting  to  the  scope  of  the  projects  if 
they  are  to  be  available  as  an  OT  test  asset  or  capability. 
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b.  Management  Approach 

REP  subprojects  are  reviewed,  analyzed,  and  prioritized  by  the  REP 
Working  Group  (WG)  and  then  submitted  through  the  CTEIP  PEM  to  the  Deputy 
Director,  Resources  Ranges  (DDR&R)  and  the  OTICC.  The  REP  WG  consists  of  the 
WG  Chairman  and  a  primary  and  alternative  representative  from  the  Army,  Navy,  Air 
Force,  and  Marine  Corps.  Also  represented  on  the  REP  WG  are  the  Joint  Interoperability 
Test  Command  (JITC)  Operational  Test  Agencies  (OTAs),  Ballistic  Missile  Defense 
Organization  (now  known  as  the  Missile  Defense  Agency  (MDA)),  and  the  Defense 
Intelligence  Agency  (DIA).  Voting  rights  are  given  only  to  those  representatives  from  an 
OTA.  The  day-to-day  management  of  the  individual  REP  is  assigned  to  a  service  or 
defense  agency  field  level  organization.  A  subproject  manager,  an  alternate,  and  a 
financial  manager  are  named.  Normal  communications  flow  through  the  CTEIP  PEM  to 
the  REP  WG  Chairman.  It  then  goes  to  the  REP  WG  members,  and  down  the  chain  of 
command  to  the  subproject  manager  at  the  field  level.  The  communications  reverse  to 
flow  up  the  chain  of  command.  [Ref.  2:pp.  2-3  to  2-4]  The  key  management  document 
for  REP  subprojects  is  the  Quad  Chart.  It  provides  a  detailed  subproject  description; 
identifies  individuals  and  organizations;  and  shows  key  milestones,  funding  sources,  and 
the  funding  stream.  [Ref.  2:p.  3-4] 


Figure  2.4.  REP  Management  Structure.  [From:  Ref.  3:p.  11] 
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D.  CTEIP  RELATED  ORGANIZATIONS 

Many  organizations  have  either  direct  oversight  responsibility,  indirect  oversight 
responsibility,  or  direct  execution  responsibility  for  the  three  CTEIP  project  areas. 

1.  Director,  Operation  Test  and  Evaluation  (DOT&E) 

The  DOT&E  is  functionally  assigned  to  the  Office  of  the  Secretary  of  Defense 
(OSD).  He  serves  as  the  principal  staff  assistant  and  senior  advisor  on  matters  related  to 
Operational  Test  and  Evaluation  (OT&E)  and  Live  Fire  Test  and  Evaluation  (LFT&E). 
DOT&E  issues  policy  and  procedures  for  DoD  OT&E  and  LFT&E;  reviews  and  analyzes 
the  results  of  testing  conducted  for  each  major  DoD  acquisition  program;  and  provides 
independent  assessments  to  the  Secretary  of  Defense,  the  Under  Secretary  of  Defense  for 
Acquisition  Technology,  and  Logistics  (USD  (AT&L))  and  Congress.  DOT&E  is 
responsible  for  budgetary  recommendations  regarding  OT&E  to  ensure  that  the  OT&E 
and  LFT&E  for  major  acquisition  programs  are  planned  so  that  the  operational 
effectiveness,  suitability,  vulnerability,  and  lethality  in  combat  use  are  tested  properly. 
DOT&E  was  also  given  the  oversight  of  the  DoD’s  Major  Range  and  Test  Facility  Base 
(MRTFB)  as  well  as  the  oversight  of  the  development  of  test  resources  such  as 
instrumentation,  targets,  and  other  threat  simulators.  This  includes  the  modeling  and 
simulation  infrastructure.  CTEIP  Program  Element  0604940D  is  directly  managed  under 
DOT&E  by  the  Deputy  Director,  Resources  and  Ranges.  [Ref.  l:p.  A-l] 
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Figure  2.5.  DOT&E  Organizational  Chart.  [After:  Ref.  l:p.  A-l], 

2.  Defense  Test  and  Training  Steering  Group  (DTTSG) 

The  DTTSG  oversees  the  integration  of  all  training  and  test  range  instrumentation 
and  the  development  of  requirements  to  facilitate  a  consolidated  acquisition  policy  for 
training  and  test  capabilities.  Embedded  test  and  training  capabilities  in  weapons  systems 
are  included  in  this  effort.  The  DDTSG  is  chartered  to  provide  direction,  policy,  and 
guidance  for  all  DoD  development  and  acquisition  programs  for  hardware  simulators, 
emitters,  software  simulations,  hybrid  representations,  and  surrogates  of  threat  weapon 
systems.  The  DTTSG  is  chaired  by  DOT&E.  The  Deputy  Director  for  Resources  and 
Ranges,  DOT&E  and  the  Director,  Readiness  and  Training,  Deputy  Under  Secretary  of 
Defense  (Readiness)  are  the  Co-Executive  Secretaries.  The  Steering  Group  itself  is  made 
up  of  members  representing  the  offices  of  Deputy  Under  Secretary  of  Defense 
(Readiness);  Deputy  Under  Secretary  of  Defense  (Science  and  Technology);  Deputy 
Director,  Strategic  and  Tactical  Systems  (Developmental  Test  and  Evaluation);  Test  and 
Evaluation  Representatives  from  the  Army,  Navy,  Air  Force,  and  Marine  Corps;  Training 
Representatives  from  the  Army,  Navy,  Air  Force,  and  Marine  Corps;  Joint  Chiefs  of 
Staff,  J-8;  Defense  Threat  Reduction  Agency;  MDA;  Defense  Intelligence  Agency;  and 
Defense  Information  Systems  Agency.  [Ref.  l:p.  A-l,  2]. 
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3.  OSD  Test  Investment  Coordinating  Committee  (OTICC) 

The  OTICC  was  chartered  in  March,  2000  as  a  result  of  combining  two  of  three 
original  committees  under  the  DTTSG.  The  committees  that  were  combined  were  the 
Test  and  Evaluation  Resource  Committee  (TERC)  and  the  CROSSBOW  Committee. 
The  OTICC  has  a  more  direct  role  of  oversight  and  management  over  the  CTEIP.  It 
functions  to  coordinate  the  working  level  activities  through  several  formal  working 
groups.  Pertinent  directly  to  the  CTEIP  oversight  role  are  the  Resource  Enhancement 
Project  Working  Group  and  the  Test  Technology  Development  and  Demonstration 
Working  Group.  The  REP  Working  Group  directly  oversees  the  Operational  Test 
subprojects  executed  under  the  REP.  The  TTD&D  Working  Group  oversees  the 
execution  of  the  TTD&D  subprojects.  The  OTICC  Chair  is  the  Deputy  Director, 
Resources  and  Ranges.  The  CTEIP  Program  Element  Manager  (PEM)  serves  as  the 
Executive  Secretary.  Membership  on  the  OTICC  is  at  the  0-6/GM-15  level  or  above  and 
are  representatives  from  the  offices  of  Assistant  Director,  Test  and  Evaluation  Resources, 
Missile  Defense  Agency;  Deputy  Director  for  Resources,  Test  and  Evaluation 
Management  Agency,  U.S.  Army;  Director  of  Test  and  Evaluation  and  Technology 
Requirements,  T&E  Infrastructure,  U.S.  Navy;  Chief,  Test  and  Evaluation  Resources  and 
Infrastructure  Division,  Test  and  Evaluation  Directorate,  U.S.  Air  Force;  Marine  Corps 
Operational  Test  and  Evaluation  Activity;  NCR  Liaison  Officer,  Joint  Interoperability 
Test  Command,  Defense  Information  Systems  Agency;  Director  for  Special  Weapon 
Technology,  Testing  Division  Defense  Threat  Reduction  Agency;  Training 
Instrumentation  Resource  Investment  Committee  (TIRIC)  (Executive  Secretary);  Threat 
Simulator  Working  Group  (Chair);  and  the  Director,  Joint  Program  Office  (T&E).  [Ref. 
l:p.  A-2] 
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Figure  2.6.  OTICC  Organization.  [From:  Ref.  3:p.  6]. 

4.  T&E  Executive  Agent  (T&E  EA) 

The  Service  T&E  Executive  Agent  was  established  in  1993.  This  organization 
provides  the  oversight  to  all  of  the  Services  RDT&E  budgets  for  investments  in  test 
capabilities  and  for  streamlining  the  T&E  infrastructure.  [Ref.  l:p.  A-3] 

a.  The  Board  of  Directors  (BoD) 

The  BoD  is  made  up  of  all  three  Service  Vice  Chiefs  of  Staff  and  the 
DOT&E.  The  Assistant  Commandant,  U.S.  Marine  Corps,  is  a  non-voting  member.  The 
Chair  of  the  BoD  is  rotated  among  each  of  the  Service  Vice  Chiefs  and  the  DOT&E 
every  two  years.  The  mission  of  the  BoD  is  to  provide  corporate  guidance  to  the  overall 
development  of  T&E  investments  without  regard  to  Service  or  Defense  Agency 
ownership.  [Ref.  l:p.  A-3] 

b.  The  Board  of  Directors  (Executive  Secretariat)  (BoD  (ES)) 

The  BoD  (ES)  membership  consists  of  all  the  T&E  Principals  among  all 
the  services.  This  includes  the  Deputy  Under  Secretary  of  the  Anny  (Operations 
Research);  the  Director,  Air  Force  Test  and  Evaluation;  the  Director,  Test  and  Evaluation 
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and  Technology  Requirements,  T&E  Infrastructure,  U.S.  Navy;  and  the  Deputy  Director 
DOT&E  (Resources  and  Ranges).  The  BoD  (ES)  reports  directly  to  the  BoD  and  the 
Chair  is  the  T&E  Principal  from  the  same  service  as  the  BoD  Chair.  The  Body  (ES)  acts 
as  the  agent  to  the  BoD  in  implementing  the  T&E  Reliance  effort  and  guiding  the  T&E 
infrastructure  investment  and  management  policy.  Approval  of  joint  T&E  requirements 
comes  from  the  BoD(ES)  as  well  as  needs  and  solutions  recommendations  for  CTEIP 
projects.  [Ref.  l:p.  A-3] 


Figure  2.7.  T&E  Executive  Agent  Structure.  [After:  Ref.  6] 

c.  The  Board  of  Directors  (Executive  Secretariat  Staff)  (BoD 
(ESS)) 

The  BoD  (ESS)  is  made  up  of  representatives  from  the  offices  of  the  T&E 
Principals.  The  Chair  is  from  the  same  Service  as  the  BoD  Chair.  The  Body  (ESS) 
supports  the  efforts  of  the  BOD  (ES)  in  recommending  T&E  infrastructure  investment 
guidance  and  management  policy.  The  BoD  (ESS)  works  closely  with  the  Test  Resource 
Advisory  Group  and  has  input  into  the  CTEIP  needs  and  solutions  selection  process. 
[Ref.  l:pp.  A-3,  4] 
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d.  Test  Resource  Advisory  Group  (TRA  G) 

The  TRAG  is  made  up  of  the  Commanding  General,  Developmental  Test 
Command;  Director  of  Operations,  Air  Force  Materiel  Command;  Department  of  the 
Navy  representative;  and  the  Deputy  Director,  DOT&E,  Resources  and  Ranges.  The 
TRAG  is  chaired  by  a  principal  from  a  service  or  organization  not  chairing  the  BoD  and 
the  BoD  (ES).  The  TRAG  reports  directly  to  the  BoD  (ES).  The  Director,  Joint  Program 
Office  (T&E)  serves  as  the  Executive  Secretary  of  the  TRAG.  The  function  of  the  TRAG 
is  to  implement  the  policies,  decisions,  and  guidance  of  the  T&E  EA.  The  TRAG 
provides  input  into  the  CTEIP  needs  and  solution  selection  process  and  offers 
recommendations  on  T&E  infrastructure  requirements  and  priorities.  [Ref.  l:p.  A-4] 

e.  The  Joint  Program  Office  for  T&E  (JPO  (T&E)) 

The  JPO  (T&E)  is  made  up  of  DOT&E  military,  civilian,  and  contractor 
personnel.  It  is  headed  by  a  Senior  Executive  Service  Director.  The  main  function  of  the 
JPO  (T&E)  is  to  facilitate  communication  for  the  T&E  EA  in  the  areas  of  T&E 
infrastructure  management  and  investment  planning.  [Ref.  l:p.  A-4] 

f  The  T&E  Reliance  Structure 

To  facilitate  a  more  corporate -based  means  of  investment  planning,  ten 
main  areas  of  T&E  have  been  identified  and  fonned  into  multi-service  Reliance  Teams. 
Reliance  Leads  for  each  of  the  ten  areas  are  designated  and  serve  as  the  prime  points  of 
contact  for  efforts  in  their  area  across  the  services.  The  Reliance  Leads  coordinate  all  the 
service  resource  requirements  within  their  area.  The  ten  Reliance  areas  are  Air  Combat; 
Land  Combat;  Sea  Combat;  Space  Combat;  Common  Range  Instrumentation;  Electronic 
Combat;  Armaments  and  Munitions;  Targets;  Test  Environments;  and  Command, 
Control,  Communications,  Computers,  and  Intelligence  (C4I).  The  provisions  for 
oversight  of  the  Reliance  areas  are  contained  in  joint-service  memorandums  of 
agreements  (MO As).  [Ref.  l:p.  A-4] 

E.  U.S.  ARMY  T&E  ORGANIZATIONS 

The  following  organizations  serve  as  the  primary  points  of  contact  for  Army 
PMOs  in  their  pursuit  of  answers  to  T&E  planning  needs.  While  there  are  various  service 
test  agencies  that  should  be  consulted  and  included  by  the  PMOs  on  any  T&E  Integrated 
Product  Team,  the  agencies  described  below  function  at  the  headquarters  and  command 
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level  and  should  be  the  first  avenue  taken  by  the  PMOs  to  address  T&E  policy  questions 
and  T&E  infrastructure  and  investment  matters. 

1.  U.  S.  Army  Test  and  Evaluation  Management  Agency  (TEMA) 

The  mission  of  TEMA  is  to  develop  and  coordinate  all  Anny  T&E  resource  and 
policy  actions  with  OSD,  Army,  Navy,  Air  Force,  other  DOD,  and  Congressional 
activities.  The  TEMA  responsibilities  as  they  interact  with  the  CTEIP  and  with  the 
PMOs  are  to  provide  centralized  T&E  management  by  establishing  and  chairing  T&E 
forums  as  required;  administer  the  Army  portion  of  the  Central  Test  and  Evaluation 
Investment  Program  (CTEIP)  and  Resource  Enhancement  Program  (REP);  provide  staff 
management  of  all  test  programs  of  interest  to  the  Chief  of  Staff  of  the  Army;  serve  as 
HQDA  coordination  agent  for  all  T&E  policy,  resource  programming,  and  related 
programmatics;  manage  the  HQDA  staffing  and  approval  process  for  TEMPs  requiring 
HQDA  approval  and  OSD  approval;  support  the  Deputy  Under  Secretary  of  the  Army  for 
Operations  Research  (DUSA(OR))  in  his  role  as  a  member  of  the  BoD  (ES)  by  serving  as 
the  Anny  T&E  representative  on  the  BOD(ES)  staff;  as  the  Anny  representative  on  the 
BoD  (ES)  staff,  provide  representation  to  the  OSD  Test  Investment  Coordinating 
Committee  (OTICC);  represent  the  Anny  Test  &Evaluation  Command  (ATEC) 
requirements  at  the  POM  reviews;  and  serve  as  the  T&E  Functional  Chief  Representative 
(FCR)  in  support  of  the  DUSA(OR)  in  his  role  as  acquisition  workforce  T&E  Functional 
Chief.  As  the  T&E  FCR,  furnish  the  T&E  Acquisition  Career  Field  representation  to  the 
Army  Acquisition  Career  Management  Functional  Working  Group  (AACM-FWG)  and 
to  the  T&E  education  oversight  committees.  Within  these  responsibilities  are  the  means 
for  TEMA  to  be  a  key  proponent  of  the  PM  in  its  entry  into  the  CTEIP  project 
formulation  and  selection  process.  [Ref.  7] 
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Figure  2.8.  TEMA  within  the  Department  of  the  Army.  [From:  Ref.  7] 
a.  Test  &  Evaluation  Managers  Committee 

The  Test  and  Evaluation  Managers  Committee  (TEMAC)  is  an  internal 
Army  committee,  which  supports  the  Army  T&E  community.  TEMAC  conducts  studies 
and  reviews  as  directed  by  senior  Army  leadership;  provides  Army  input  to  DoD  T&E 
strategy  and  action  plans;  coordinates  among  TEMA,  PEO/PMs,  and  Army  Research, 
Development,  and  Engineering  Centers;  and  serves  as  a  forum  for  the  review  and  study 
of  issues  raised  by  the  acquisition  community.  TEMAC  is  chaired  by  TEMA.  The 
official  charter  for  the  TEMAC  was  renewed  at  the  end  of  AY  00  for  two  more  years. 
The  committee  meets  semi-annually  to  discuss  relevant  T&E  initiatives  as  well  as  to 
continue  to  work  various  T&E  issues  as  requested  by  the  Director  of  TEMA.  Each 
member  activity  shall  identify  a  T&E  manager  for  membership  to  the  TEMAC.  The 
T&E  manager  is  assigned  the  responsibility  as  the  central  point  of  contact  for  all  T&E 
matters  between  that  organization  and  Fleadquarters,  DA,  as  represented  by  TEMA.  The 
member  activities  that  comprise  the  TEMAC  are  U.S.  Army  Test  and  Evaluation 
Management  Agency,  Office  of  the  Chief  of  Staff  (TEMA)  (Chair);  U.S.  Army  Materiel 
Command  (AMC)  (Executive  Secretary);  U.S.  Army  Aviation  and  Missile  Command 
(AMCOM);  U.S.  Army  Soldier  and  Biological  Chemical  Command  (SBCCOM);  U.S. 
Army  Communications-Electronics  Command  (CECOM);  U.S.  Army  Medical  Command 
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(MEDCOM);  U.S.  Army  Test  and  Evaluation  Command  (ATEC);  U.S.  Army 
Developmental  Test  Command  (DTC);  U.S.  Army  Research  Laboratory  (ARL);  U.S. 
Army  Simulation,  Training,  and  Instrumentation  Command  (STRICOM);  U.S.  Soldier 
Support  Command  (SSC);  U.S.  Space  and  Missile  Defense  Command  (SMDC);  U.S. 
Army  Tank- Automotive  and  Armaments  Command  (TAACOM);  U.S.  Army  Training 
and  Doctrine  Command  (TRADOC);  other  agencies  as  may  be  required  to  include  Office 
of  the  Assistant  Secretary  of  the  Army,  Research  Development  and  Acquisition 
(OASARDA);  Office  of  the  Director  for  Information  Systems  for  Command,  Control, 
Communications,  and  Computers  (ODICS4);  Office  of  the  Deputy  Chief  of  Staff  for 
Operations  and  Plans  (ODCSOPS);  Office  of  the  Deputy  Chief  of  Staff  for  Logistics 
(ODCSLOG);  Office  of  the  Deputy  Chief  of  Staff  for  Intelligence  (ODCSINT);  Office  of 
the  Chief  of  Engineers  (OCOE);  Office  of  the  Surgeon  General  (OTSG);  U.S.  Army 
Industrial  Operations  Command  (IOC);  and  U.S.  Army  Materiel  Systems  Analysis 
Activity  (AMSAA).  [Ref.  7] 


Figure  2.9.  TEMAC  Membership  Across  the  U.S.  [From:  Ref.  7] 
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2.  U.  S.  Army  Test  and  Evaluation  Command  (ATEC) 

The  ATEC  headquarters  has  the  oversight  of  the  Army  piece  of  the  Test 
Investment  Planning  process.  One  of  ATEC’s  subordinate  commands,  the 
Developmental  Test  Command  (DTC),  plays  an  active  role  in  the  Needs  and  Solutions 
process.  ATEC  participates  on  each  of  the  Reliance  panels  and  provides  the  Anny  Needs 
and  Solutions  on  instrumentation  and  range  infrastructure  requirements.  ATEC 
headquarters  has  the  final  submission  approval  for  all  Anny  recommended  needs  and 
solutions.  In  addition,  ATEC  is  responsible  for  providing  POM  inputs  for  Anny  T&E 
requirements.  Those  requirements  are  submitted  through  TEMA,  who  is  responsible  for 
representing  the  Army  T&E  requirements  at  all  POM  reviews. 

3.  Program  Manager,  Instrumentation,  Targets,  and  Threat  Systems 
(PM,  ITTS) 

PM,  ITTS  is  responsible  for  managing  the  research,  development,  design, 
acquisition,  fielding,  modification,  and  capability  accounting  of  major  instrumentation, 
targets,  and  threat  simulators  required  for  developmental  and  operational  test  and 
evaluation  (T&E)  and  training.  PM,  ITTS,  is  responsible  for:  the  management  of  the 
Central  Test  and  Evaluation  Investment  Program  (CTEIP)  and  Resource  Enhancement 
Program  (REP)  projects  for  the  Anny;  management  operations  of  targets  for  T&E  and 
training  of  Army  and  Foreign  Military  Sales  (FMS)  customer  troops;  and  management  of 
the  Army  Instrumentation,  Targets,  and  Threat  Simulators  (ITTS)  Long  Range  Planning 
Process.  PM,  ITTS  develops  and  implements  policy  direction  and  control  over  funding 
and  execution  of  major  instrumentation,  targets  and  threat  simulator/simulation  projects. 
PM,  ITTS  serves  as  the  Army's  single  manager  for  acquiring  targets,  threat 
simulators/simulations,  and  major  test  instrumentation. 
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III.  DATA  SUMMARY 


A.  INTRODUCTION 

This  chapter  presents  the  data  collected  to  answer  the  primary  and  subsidiary 
thesis  research  questions.  The  primary  modes  of  data  collection  were  literature  searches 
and  interviews.  Literature  searches  were  conducted  on  the  CTEIP  guidance  documents  to 
include  Congressional  language,  program  execution  guides,  annual  reports,  program 
briefings,  and  Internet  websites.  Interviews  were  conducted  with  the  PMO  Test  Division 
Chiefs  and  the  Senior  Test  Team  Leads.  These  interviews  were  conducted  in  person,  by 
telephone,  and  by  Internet.  Interviews  were  also  conducted  with  key  people  in  the  CTEIP 
management  and  oversight  chain. 

1.  PMO  Interviews 

The  interviews  with  the  PMO  Test  Division  Chiefs  and  the  PMO  Senior  Test 
Team  Leads  were  the  primary  method  of  addressing  the  primary  research  questions.  The 
PMOs  were  limited  to  those  Anny  PMOs  located  in  and  around  Redstone  Arsenal,  AL. 
The  PMOs  represented  in  the  interviews  include  those  within  the  Program  Executive 
Office  (PEO),  Aviation;  PEO  Tactical  Missiles;  PEO  Air  and  Missile  Defense;  PEO, 
Intelligence,  Electronic  Warfare  &  Sensors;  and  Missile  Defense  Agency  (MDA). 
Specific  names  and  offices  will  not  be  identified  or  attributed  to  specific  comments, 
although  anonymity  was  not  requested. 

2.  CTEIP  Management  Interviews 

Interviews  were  conducted  with  key  positions  within  the  CTEIP  management  and 
oversight  agencies.  The  interviewees  were  primarily  in  Anny  and  OSD  level  offices. 
These  personnel  have  approximately  two  to  twelve  years  experience  in  their  current 
positions.  They  also  have  multiple  years  of  experience  in  the  T&E  arena  serving  in  T&E 
service  agencies  and  in  OSD  level  positions  and  actively  participating  at  all  levels  of  the 
T&E  Investment  process.  Again,  specific  offices  and  names  were  not  linked  with 
specific  comments  although  this  was  not  requested. 

3.  Literature  Search 

The  literature  search  was  conducted  to  review  the  current  CTEIP  documentation. 
Program  Execution  Guides  and  Annual  Reports  were  reviewed  to  determine  the  extent  to 
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which  PMO  have  participated  in  CTEIP  projects  now  and  in  the  past.  The  investment 
process  was  reviewed  to  determine  the  ease  with  which  a  PMO  could  interpret  and 
navigate  the  path  toward  CTEIP  project  proposal  submission.  Finally,  in  this  time  of 
electronic  access,  an  Internet  search  was  conducted  to  determine  the  extent  to  which 
CTEIP  information  existed  on  the  Internet  and  again,  how  easily  a  PMO  could  utilize  the 
Internet  information  to  prepare  and  submit  a  CTEIP  project  proposal. 

C.  DATA  PRESENTATION 

This  section  presents  the  research  information  obtained  to  address  the  primary  and 
subsidiary  research  questions.  In  order  to  detennine  how  a  PMO  could  become  more 
involved  in  the  CTEIP  process,  it  is  necessary  to  define  the  processes  themselves.  To 
that  end,  a  description  of  the  DoD-level  Test  Investment  Process  is  presented.  After  a 
review  of  the  overall  test  investment  structure,  the  specific  processes  of  the  CTEIP  and  its 
three  projects  the  JIM,  REP,  and  TTD&D,  are  also  defined.  A  review  of  these  processes 
provides  information  to  address  the  question,  “How  can  a  Program  Management  Office 
insert  themselves  into  the  CTEIP  needs  and  solution  process  in  order  to  obtain  funding 
for  test  program  developments?”  Next  in  the  section  is  a  presentation  of  the  results  from 
the  interviews.  This  information  is  valuable  in  answering  the  primary  question,  “Would 
funding  from  CTEIP  be  of  value  to  the  PMOs  test  program?”  All  the  research  obtained 
contributed  to  the  overall  analysis. 

1.  Test  Investment  Process 

To  understand  the  CTEIP  Investment  process  one  must  first  have  information  on 
the  overall  Test  Investment  process.  The  DoD  Test  Investment  Review  process  is  mainly 
interested  in  investments  that  have  costs  of  $1  million  or  more  in  a  single  year,  or  $5 
million  over  the  total  project  development.  There  are  two  categories  of  investments 
covered  by  this  process:  those  that  are  service-unique,  and  those  that  are  Joint  Service 
investments.  The  service-unique  investments  are  funded  by  the  services  themselves.  The 
Joint  Service  investments  may  be  funded  by  a  combination  of  funds  from  the  services 
involved  in  the  project  or  through  an  approved  CTEIP  project.  The  review  process  for 
service-recommended  investments  consists  of  the  steps  illustrated  by  Figure  3.1.  CTEIP- 
funded  projects  are  approved  by  a  separate  review  process  and  will  be  addressed  later  in 
this  chapter.  [Ref.  8] 
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The  Joint  Program  Office  for  Test  and  Evaluation  (JPO  (T&E))  initiates  the 
process  with  a  Needs  call  to  the  service  Headquarters.  This  call  is  for  test  and  evaluation 
investment  needs  to  address  known  test  capability  shortfalls  within  the  Service 
framework.  The  services  then  promulgate  the  call  using  their  individual  processes,  and 
Need  statements  are  generated.  Each  service  then  collects,  validates,  and  integrates  their 
test  resource  Needs  and  submits  them  to  the  JPO  (T&E).  [Ref.  8] 
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Figure  3.1.  The  DoD  Test  Investment  Process.  [From  Ref.  8] 


Upon  receipt  of  each  service's  list  of  Needs,  the  JPO  groups  them  into  Reliance 
areas  and  parcels  them  out  to  the  appropriate  Reliance  Leads.  Under  the  management  of 
the  Leads,  Reliance  Panels  evaluate  the  Needs  and  recommend  endorsement/non¬ 
endorsement.  The  Panels  also  seek  opportunities  to  combine  Needs  and  resolve  conflicts. 
The  lists  are  then  returned  to  the  JPO,  which  performs  the  administrative  function  of 
collecting  the  lists  and  distributing  them  to  the  Test  and  Evaluation  Reliance  Investment 
Board  (TERIB).  [Ref.  8] 

The  TERIB's  primary  functions  in  the  process  is  to  deconflict  across  Reliance 
areas  and  to  separate  those  projects  that  may  qualify  for  joint  funding  from  those  that 
meet  only  single  Service  needs.  The  joint  Needs  are  prioritized  and  forwarded  to  the 
Board  of  Operating  Directors  (BoOD).  [Ref.  8] 
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After  review,  the  BoOD  forwards  all  Needs  to  the  BoD  and  the  Defense  Test  and 
Training  Steering  Group  (DTTSG).  The  Test  and  Evaluation  Resource  Committee 
(TERC)  integrates  the  service  joint  Needs  for  which  CTEIP  funding  is  sought  with 
Defense  Agency  Needs,  and  forwards  the  integrated  CTEIP  Needs  to  the  DTTSG.  The 
DTTSG  and  the  BoD  develop  constraints  for  Solutions.  Initial  inputs  to  the  Defense 
Planning  Guidance  are  provided  to  the  DTTSG.  [Ref.  8] 

Through  the  DTTSG,  OSD  (DOT&E)  reviews  both  CTEIP  and  Service  Needs.  It 
then  issues  any  constraints  or  guidance  to  be  used  in  the  development  of  Solutions  to 
satisfy  those  Needs  endorsed  by  the  BoD  (for  Service)  and  the  TERC  (for  CTEIP).  Once 
the  OSD  endorses  the  Needs  list  and  issues  constraints  and  guidance,  the  JPO  (T&E) 
issues  a  Solutions  call  for  Service  Solutions,  and  the  TERC  issues  a  call  for  CTEIP 
Solutions.  The  Services  then  respond  in  a  process  very  similar  to  that  used  for  the  Needs 
Call.  The  Solutions  are  collected,  grouped,  and  distributed  to  cognizant  Leads.  The 
Reliance  panel  reviews  the  Solutions  for  endorsement/non-endorsement,  and  after  they 
are  deconflicted  and  integrated,  they  are  submitted  to  the  TERIB  and  then  to  the  BoOD 
and  the  BoD.  [Ref.  8] 

The  Test  Investment  Planning  and  Review  process  hinges  upon  the  production  of 
a  Test  Capability  Master  Plan  (TCMP)  for  each  Reliance  area.  This  document  is 
intended  to  define  T&E  efforts  within  a  Lead  area.  It  describes  the  scope  of  the  Reliance 
area,  test  methodology,  existing  capabilities,  and  projected  test  capability  requirements 
(investments).  It  also  provides  the  overall  direction  and  architecture  for  the  Lead  area. 
The  TCMP  is  the  vehicle  by  which  Reliance  evaluates  new  Needs  and  Solutions  for  their 
congruency  with  planned  efforts  and  the  investment  strategy.  Each  Lead  uses  the  results 
of  the  Needs  and  Solutions  call  to  update  the  previous  year's  TCMP  to  reflect  new 
initiatives,  strategies,  and  areas  of  emphasis.  Each  area  TCMP  is  forwarded  to  the 
TERIB.  [Ref.  8] 

The  TERIB  uses  the  individual  TCMPs  and  endorsed  Solutions  to  develop  two 
documents:  the  Test  Resource  Master  Plan  (TRMP),  and  the  Test  Investment  Strategy 
(TIS).  These  documents  are  intended  to  serve  as  blueprints  to  define  and  guide  the 
Service-wide  test  investments.  The  TIS  provides  a  concise  statement  of  the  long-term 
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objectives  to  be  gained  by  these  investments.  The  TRMP  serves  as  the  road  map  to  attain 
the  TIS's  vision  and  includes  an  integrated  and  prioritized  compendium  of  individual 
projects  intended  to  execute  the  plan.  The  actual  process  by  which  Reliance  develops  the 
TRMP  and  the  TIS  includes  various  feedback  mechanisms  whereby  the  TERIB  can 
confer  with  Reliance  Leads  to  negotiate  acceptable  compromises.  If  participants  at  the 
working  level  cannot  reach  agreement,  the  process  allows  unresolved  issues  to  be  carried 
forward  to  higher  authority  (BoOD,  BoD,  or  ultimately  DOT&E).  Upon  completion  of 
the  TRMP  and  the  TIS,  the  TERIB  sends  the  documents  to  the  BoOD  and  then  the  BoD. 
Through  DOT&E,  OSD  approves  the  final  TRMP.  This  ensures  that,  in  addition  to 
serving  to  communicate  a  common  investment  strategy  to  the  T&E  community,  the 
TRMP  and  TIS  are  used  by  OSD  in  its  development  of  the  Defense  Planning  Guidance. 
Thus,  OSD  and  the  Services  will  consider  T&E  investment  priorities  in  their  individual 
Program  Objective  Memorandums  (POMs).  [Ref.  8] 

2.  CTEIP  Investment  Process 

The  investment  planning  process  for  CTEIP  is  a  joint  process  and  is  integrated 
into  the  overall  DoD  T&E  investment  planning  process  that  was  described  in  the  previous 
section.  The  overall  DoD  T&E  investment  planning  process  is  synchronized  with  the 
Planning,  Programming,  and  Budgeting  System  (PPBS).  The  total  objective  of  the 
CTEIP  process  is  to  have  a  set  of  joint  T&E  investments  of  high  priority  identified, 
reviewed,  evaluated,  and  approved  so  that  the  funding  profiles  for  the  approved  projects 
can  be  included  into  each  biannual  (POM).  The  investment  planning  for  DoD  T&E  is  a 
continuing  process  of  detennining  test  requirements  (Needs)  and  selecting  specific  test 
investments  (Solutions)  that  will  alleviate  the  shortfalls  in  test  capabilities.  [Ref.  1  :p.  4] 

The  OTICC  is  the  advisory  body  for  DOT&E  for  the  oversight  and  management 
of  CTEIP.  An  integrated  CTEIP  planning  process  is  in  place.  Participation  in  this 
process  by  the  T&E  Executive  Agents  (T&E  EA),  the  services,  and  the  Defense  Agencies 
is  critical  to  responding  to  the  PPBS  timetable.  This  timetable  is  described  in  Table  3.1. 
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Months 

Fiscal 

Year 

Activities 

Feb 

XX-3 

T&E  EA  issues  FYXX  Needs  and  Solutions  Call. 

Jun 

XX-3 

Services  submit  Needs  and  Solutions 

DOT&E  forwards  Defense  Agency  Needs  and  Solutions  to 
T&E  EA. 

Jan 

XX-2 

Defense  Planning  Guidance  (DPG)  issued. 

Feb-May 

XX-2 

POM  preparation  for  FYXX  to  FYXX+5. 

Mar 

XX-2 

T&E  EA  forwards  deconflicted  and  prioritized  CTEIP 
proposals  to  DOT&E. 

May 

XX-2 

DOT&E  resources  proposals  and  includes  results  as  CTEIP 
input  to  the  POM. 

Jun 

XX-2 

DOT&E  submits  CTEIP  input  to  FYXX  POM. 

Jan-Feb 

XX- 1 

FYXX  President’s  Budget  (PB)  submitted  to  Congress. 

Mar- Sep 

XX- 1 

Congressional  enactment  of  FYXX  Defense  Appropriations. 

Oct 

XX 

FYXX  funding  approved. 

Table  3.1.  Integrated  Test  Investment  and  PPBS  Process.  [From:  Ref.  l:p.  5] 

Although  the  process  is  continuous,  there  is  a  beginning  point.  Every  two  years, 
the  T&E  EA  issues  a  Needs  and  Solutions  Call  to  the  services  T&E  headquarters,  who,  in 
turn,  disseminate  it  to  their  service  T&E  agencies.  It  is  also  sent  to  DOT&E  who  forward 
it  to  the  participating  Defense  Agencies.  This  is  an  organized  means  of  giving  the  entire 
DoD  community  the  opportunity  to  submit  their  test  capability  investment  requirements. 
For  any  given  POM  fiscal  year,  the  Call  is  issued  three  years  before  the  appropriation. 
The  Defense  Agencies  then  forward  their  CTEIP  Needs  and  Solutions  to  DOT&E,  who 
forwards  them  to  the  T&E  EA  for  deconfliction,  elimination  of  duplication,  and  possible 
consolidation  with  like  service  Solutions.  Those  Solutions  that  have  the  potential  of 
meeting  the  CTEIP  criteria,  mainly  that  of  jointness,  are  prepared  as  CTEIP  proposals, 
prioritized,  and  submitted  to  DOT&E  for  consideration  as  a  validated  CTEIP  project  and 
incorporation  into  the  POM.  [Ref.  l:pp.  4-5] 

3.  CTEIP  Project  Selection  and  Execution 

a.  Joint  Improvement  and  Modernization  (JIM)  Projects 
The  CTEIP  JIM  projects  fall  out  of  the  overall  T&E  Investment  Process 
and  the  CTEIP  planning  process  described  in  the  sections  above.  Key  to  obtaining  an 
approved  CTEIP  project  for  inclusion  in  the  POM  is  the  preparation  of  a  proposal.  The 
JIM  proposal  should  clearly  demonstrate  the  benefits  to  the  T&E  community  through  the 
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execution  of  this  particular  Solution.  It  should  also  specify  clearly  that  it  be  in  answer  to 
a  valid  Need. 

The  evaluation  of  the  proposal  takes  place  at  each  level  of  the  process 
within  the  services,  the  Defense  Agencies,  and  within  the  T&E  EA.  The  proposal  should 
be  able  to  clearly  answer  questions  that  reflect  the  Congressionally  set  criteria  of  CTEIP. 
Is  the  requirement  valid?  Is  it  generic  or  reflective  of  a  specific  program?  Can  the 
requirement  be  satisfied  in  another  way?  Does  the  proposed  capability  have  multi¬ 
service  or  joint  application?  Are  there  sufficient  potential  users  of  the  proposed 
capability  to  justify  the  development  expense?  Will  the  project  be  interoperable  with 
other  DoD  facilities?  Are  the  risks  acceptable?  [Ref.  l:p.  6] 

After  receipt  of  the  proposals  from  the  services,  T&E  EA  will  send  a 
prioritized  list  of  CTEIP  proposals  to  DOT&E.  This  list  will  be  used  to  build  the  CTEIP 
input  into  the  POM.  At  any  given  time  there  are  continuing  projects  in  various  phases  as 
well  as  proposed  new  start  projects.  The  funding  levels  for  the  new  start  projects  will  be 
dependent  on  the  funding  needs  of  the  ongoing  projects  and  on  the  fiscal  guidance  given 
for  the  POM  years.  [Ref.  l:p.  6]  The  total  CTEIP  POM  submission  is  then  submitted  to 
the  OSD  Comptroller.  If  a  project  is  not  considered  for  the  first  two  years  of  the  POM,  it 
will  not  be  considered  further.  Continued  consideration  will  only  occur  with 
resubmission  through  the  Needs  and  Solutions  Calls.  [Ref.  l:p.  7] 

Once  a  project  is  identified  as  a  JIM  project,  the  CTEIP  PEM  prepares  a 
Test  Package  Directive  (TPD)  that  is  sent  to  the  Service  or  Defense  Agency  OTICC 
member.  The  TPD  initiates  Phase  I  of  the  project  and  directs  the  preparation  of  a  Test 
Capability  Requirements  Document  (TCRD),  a  draft  Life  Cycle  Support  Plan  (LCSP), 
and  a  Program  Management  Plan  (PMP)  to  include  an  Analysis  of  Alternatives  (AoA)  as 
an  annex.  In  the  TPD,  the  executing  service  or  Defense  Agency  is  identified.  The  JIM 
projects  are  then  executed  by  organizations  with  the  required  technical  expertise, 
management  structure,  facilities,  and  support  elements  within  the  designated  Service  or 
Defense  Agency.  Overall  management  oversight  is  always  provided  by  the  OTICC  and 
the  CTEIP  PEM;  however,  day-to-day  management  is  left  to  the  individually  named 
Project  Director  (PD).  [Ref.  l:p.  7] 
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The  transition  from  Phase  I  to  Phase  II  is  based  on  the  completion  of  the 
actual  concept  development  and  the  completion  and  approval  of  the  required 
documentation.  During  Phase  II,  the  TPDs  are  prepared  annually  for  signature  of  the 
CTEIP  PEM.  Also  required  of  the  executing  PD  are  a  PMP  submitted  annually;  monthly 
status  reports;  annual  Mid-Year  Review  briefings;  program  reviews  as  deemed  necessary 
for  the  development  process  and  as  requested  by  the  OTICC;  and  a  final  report  at  project 
completion.  Financial  management  reporting  is  required  as  a  part  of  the  monthly  status 
reports,  all  reviews,  the  Mid-Year  Review  briefing,  and  the  final  report.  [Ref.  l:pp.  10- 
13] 

b.  Test  Technology  Development  and  Demonstration  (TTD&D) 
Projects 

The  TTD&D  project  and  subproject  nomination  process  begins  with  the 
issuance  of  a  Call  for  Proposal  memorandum  signed  by  the  TTD&D  Project  Director. 
This  usually  takes  place  in  February  of  each  year  with  the  subprojects  to  begin  the  next 
fiscal  year.  Proposals  are  submitted  to  DOT&E  through  the  service  or  agency  formal 
chain  of  command  to  the  service/agency  coordinator  and  the  OTICC  representative.  The 
service/agency  coordinator  and  the  OTICC  representative  are  responsible  for  reviewing 
and  prioritizing  the  proposals  prior  to  submitting  them  to  DOT&E.  They  are  then 
forwarded  to  the  TTD&D  Working  Group  for  review.  [Ref.  5:p.  3-1] 

The  selection  criteria  for  the  TTD&D  subprojects  are  based  on  the 
national  T&E/Training/Warfighting  priorities;  DoD  guidance  and  policy;  evolving 
technologies;  and  future  forecasts  of  Warfighting  capabilities  and  needs.  On  an  annual 
basis,  the  TTD&D  PD  will  submit  to  the  DDOT&E/RR,  through  the  CTEIP  PEM,  a  list 
of  recommended  subprojects  for  the  coming  year.  A  Letter  of  Instruction  (LOI)  will  be 
prepared  to  provide  specific  direction  on  the  project  execution.  Monthly  status  reports 
are  submitted  by  the  PD  for  each  funded  subproject,  to  include  obligation  and 
expenditure  data.  Annual  assessment  reports  are  submitted  through  the  service/agency 
coordinator  no  later  than  the  end  of  August.  Release  of  next  year  funding  is  contingent 
on  the  receipt  of  this  report.  A  Mid-Year  Review  briefing  will  be  prepared  on  the 
progress  and  status  of  the  subprojects.  These  are  not  presented  individually  at  the  CTEIP 
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Mid-Year  Review,  but  a  synopsis  of  the  entire  TTD&D  project  is  given  based  on  the 
individual  briefings.  [Ref.  5:pp.  4-1  to  4-4] 


Congressional  language  in  2001  established  the  Test  and  Evaluation, 
Science  and  Technology  (T&E/S&T)  Program  Element.  This  new  program  began  in  FY 
02  and  focuses  on  transitioning  technologies  from  the  laboratories,  academia,  and 
industry  to  T&E  capability  developers.  The  objectives  of  the  new  program  were  so 
similar  to  TTD&D  that  there  was  concern  of  the  efforts  being  deemed  duplicative, 
therefore,  the  decision  was  made  by  DOT&E  to  begin  phasing  TTD&D  out  as  a  distinct 
CTEIP  project  beginning  in  FY  03.  Ongoing  projects  will  be  completed  with  the  last 
year  of  execution  being  FY  04.  Discussion  on  the  TTD&D  was  left  in  this  thesis  to 
provide  information  to  the  PMO.  The  selection  criteria  for  the  S&T  programs  will  be 
similar  to  that  described  within  this  document. 

c.  Resource  Enhancement  Projects  (REP) 

Preliminary  Test  and  Evaluation  Master  Plans  (TEMP)  for  major 
acquisition  programs  are  submitted  to  OSD  prior  to  the  Milestone  A  review.  Estimated 
key  OT  resource  requirements  are  defined  within  the  TEMP  to  the  extent  known.  Known 
resource  shortfalls  should  also  be  defined  in  the  TEMP.  In-cycle  REP  subproject 
proposals  typically  support  OT  efforts  in  response  to  the  known  TEMP  defined  shortfalls. 
At  times,  REP  proposals  in  answer  to  previously  unknown  OT  requirements  are 
presented,  usually  out  of  the  normal  project  selection  cycle.  These  later  REP  proposals 
are  the  prime  candidates  for  the  short  time  frame  of  the  REP  projects.  [Ref.  2:p.  3-1] 

Nominations  for  REP  projects  may  originate  from  the  service  or  defense 
agency  representatives,  but  only  the  REP  WG  voting  members  can  formally  submit  the 
projects  for  REP  funding  consideration.  The  REP  WG  then  reviews  and  prioritizes  the 
nominated  subprojects  and  submits  the  list  to  the  CTEIP  PEM.  The  Deputy  Director 
(R&R)  coordinates  with  OTICC  to  make  the  final  detennination  and  recommendation  to 
DOT&E.  The  list  of  DOT&E  approved  subprojects  will  become  the  approved  REP  for 
that  given  fiscal  year.  [Ref.  2:p.  3-2] 

There  are  specific  criteria  that  the  candidate  subprojects  must  meet  before 
they  can  be  accepted  for  REP  funding.  The  subproject  must  not  require  more  than  three 
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fiscal  years  of  funding.  The  subproject  must  resolve  a  documented  OT  shortfall  within 
three  years  of  the  project’s  start.  The  shortfall  must  be  consistent  with  ongoing 
formalized  OT  planning  as  reflected  in  an  approved  TEMP  or  other  approved  OT&E 
planning  source  documents.  The  subproject  must  support  a  milestone  decision  or  a  major 
development  decision  on  a  system,  to  include  major  upgrades,  within  the  next  five  years. 
The  subproject  must  have  a  firm  completion  date  prior  to  the  first  documented  OT  need 
date.  The  subproject  must  be  executable.  In  addition  to  the  project  criteria,  there  are 
constraints  on  the  funding  received  from  REP.  REP  funds  may  not  be  used  for  the 
conduct  of  OTs  or  for  Operation  and  Maintenance  (O&M)  costs  that  occur  after  the  initial 
installation.  The  manufacture  or  procurement  of  multiple  copies  of  a  capability  is  not 
allowed  unless  the  duplication  is  specifically  required  to  resolve  a  documented  shortfall. 
[Ref.  2:p.  3-7] 


Figure  3.2.  REP  Subproject  Nomination  and  Approval  Process.  [From:  Ref.  2:p.  3-2] 


It  is  recognized  by  the  REP  WG  that  unforeseen  OT&E  issues  arise  at  any 
time  during  the  year.  Any  proposal  considered  out-side  the  normal  PPBS  timeline  is 
referred  to  as  an  out-of-cycle  nomination.  The  REP  WG  will  consider  these  out-of-cycle 
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nominations  at  any  time  during  the  year.  There  are  guidelines  for  submitting  an  out-of¬ 
cycle  proposal.  The  full  nomination  package  must  be  prepared  and  submitted  to  the  WG 
member  representing  OTA.  Time  at  an  upcoming  REP  WG  meeting  is  requested  by  the 
WG  member  in  order  to  address  the  special  request.  It  is  then  reviewed  and  prioritized  as 
the  other  nominations  were.  If  the  request  is  received  prior  to  the  REP  PMP  being 
finalized,  then  all  the  candidates  are  re -prioritized  to  include  the  new  request.  If  the 
nomination  is  received  after  the  REP  PMP  is  issued  for  a  given  FY,  then  the  REP  WG 
will  review  it  and  an  appropriate  course  of  action  will  be  recommended  through  the 
CTEIP  PEM  to  the  Deputy  Director  (R&R)  and  the  OTICC.  The  recommendation  will 
address  priority  and  any  necessary  reprogramming  of  funds  or  the  need  to  seek  additional 
funds  from  OSD.  The  executing  service  or  defense  agency  OTICC  Principal  must  concur 
before  any  REP  subproject  loses  funds  due  to  the  approval  of  an  out-of-cycle  submission. 
[Ref.  2:p.  3-11] 

A  PMP  is  prepared  by  the  REP  WG  chairman  in  response  to  the  TPD 
issued  by  the  CTEIP  PEM.  The  approved  PMP  represents  the  project  baseline.  It 
contains  the  subproject  Quad  Charts,  consolidated  milestones  and  schedules,  and  O&E 
plan  summary.  The  TPD  contains  guidance  against  which  the  subproject  manager  will 
execute  the  subproject.  In  response  to  the  TPD,  the  subproject  manager  will  prepare  a 
Subproject  Management  Plan  (SMP)  and  submit  to  the  REP  WG  member  and  OTICC 
Principal.  The  approved  SMP  becomes  the  contract  against  which  the  subproject’s 
progress  will  be  measured.  [Ref.  2:p.  4-2] 

Funds  for  the  subproject  are  released  by  the  CTEIP  PEM  to  the  designated 
REP  Financial  Manager.  Obligations,  expenditures,  disbursements,  and  accruals  are 
tracked  by  the  REP  Financial  Manager  and  reported  in  the  monthly  status  reports  and  in 
any  subproject  reviews.  All  REP  subprojects  are  reviewed  at  least  twice  annually.  The 
first  review  takes  place  in  a  joint  meeting  between  the  OTICC  Principals  and  the  REP 
WG  in  preparation  for  the  CTEIP  Mid-Year  Review.  The  second  review  takes  place  on¬ 
site  or  near  the  location  of  the  subproject  execution.  [Ref.  2:p.  4-4] 

4.  Interviews  with  PMO  Senior  Test  Personnel 

The  personnel  interviewed  for  this  research  are  all  senior  team  leads  and  division 

chiefs  within  the  test  area  of  the  PMO  at  the  GS-14/15  level  or  equivalent.  They  are  in 

33 


positions  classified  as  being  Acquisition  Critical  and  are  members  of  the  U.  S.  Army 
Acquisition  Corps.  They  are  all  Acquisition  Corps  Certified  at  Level  III  in  the  areas  of 
Test  and  Evaluation;  Systems  Planning,  Research,  Development  and  Engineering;  and/or 
Program  Management.  They  had  all  obtained  their  certification  through  Government 
training  courses,  generally  sponsored  by  the  Defense  Acquisition  University.  This 
requires  undergoing  a  certification  process  based  on  training  in  the  required  field,  or  work 
experience  of  a  defined  length  of  time.  Level  III  Certification  in  the  area  of  Test  and 
Evaluation  requires  three  separate  classes  for  a  total  of  160  hours  of  classroom  training  in 
the  specifics  of  Test  and  Evaluation  Management  within  the  acquisition  process.  They 
each  also  have  approximately  5-10  years  or  more  of  PMO  experience. 

A  majority  of  the  interviewees  indicated  a  minimal  or  total  lack  of  knowledge  of 
the  CTEIP,  its  programs,  or  its  potential  value  to  their  test  program.  When  a  brief 
explanation  of  CTEIP  was  provided,  the  test  personnel  indicated  that  this  would  be  of 
value  to  their  program  and  they  would  be  interested  in  learning  more  about  CTEIP.  Most 
indicated  they  did  not  remember  the  topic  of  CTEIP  being  a  part  of  the  certification 
training  courses.  The  few  interviewees  that  did  have  knowledge  of  CTEIP  also  had 
extensive  experience  with  sponsoring  and  managing  CTEIP  projects  both  within  their 
PMO  as  well  as  within  prior  test  agencies.  They  indicated  that  their  CTEIP  knowledge 
was  not  a  result  of  acquisition  training,  but  of  on-the-job  involvement  with  CTEIP; 
attendance  at  T&E  sponsored  conferences;  and  continual  coordination  and  contact  with 
personnel  in  the  T&E  community.  For  those  that  had  managed  CTEIP  projects  for  their 
PMOs,  they  described  CTEIP  involvement  being  extremely  valuable  to  the  PMO  test 
program  success.  They  indicated  that  they  would  prefer  to  manage  the  CTEIP  project 
themselves,  within  the  PMO,  rather  than  indirectly  through  a  test  agency.  Maintaining 
control  of  the  CTEIP  project  helped,  in  their  mind,  to  ensure  that  their  test  program 
stayed  within  their  control  as  well.  They  also  indicated  that  the  management 
requirements  of  CTEIP  projects  were  very  time  intensive,  requiring  at  least  half  a  man- 
year  of  effort  and  usually  a  full  man-year.  This  was  hard  for  the  PMOs  to  commit  to,  as 
the  PMO  resources  are  usually  very  tight  over  a  broad  area  of  responsibilities.  When  one 
of  the  PMOs  tried  to  get  information  on  CTEIP  points  of  contact  to  establish  how  they 
could  get  involved,  they  went  to  the  Internet  as  part  of  their  search.  They  found  the 
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Internet  infonnation  slightly  confusing,  but  very  helpful.  However,  the  contact 
information  was  outdated  with  incorrect  names,  phone  numbers,  and  email  addresses 
requiring  further  searching  before  they  could  establish  contact. 

5.  Interviews  with  CTEIP  Management 

The  personnel  interviewed  for  this  research  are  ah  senior  level  T&E  experts  at  the 
GS-14/15  level  or  equivalent.  They  have  ah  worked  their  way  up  through  the  ranks  of 
test  agencies  of  a  variety  of  services.  They  have  been  involved  in  the  T&E  investment 
process  for  5-10  years  and  with  CTEIP,  specifically,  for  at  least  3  years.  Most  have 
significantly  more  time  in  this  field.  They  currently  are  working  at  the  Pentagon  in 
service  headquarters  level  offices  or  OSD  oversight  offices.  They  work  closely  with  the 
policy  and  decision-makers  within  the  T&E  arena.  These  interviewees  are  not  ah  part  of 
the  Acquisition  Corps  and  do  not  ah  have  a  requirement  for  acquisition  certification  as 
part  of  their  responsibilities. 

The  CTEIP  management  interviewees  indicated  a  definite  value  to  both  CTEIP 
and  the  PMOs  if  the  PMO  was  more  involved  in  the  CTEIP  process.  The  PMO  would 
obtain  value  by  getting  required  test  assets  and  capabilities  faster  and  for  lower  cost. 
CTEIP  would  benefit  by  having  valid  partners  in  their  developments  with  valid 
requirements  for  the  test  investments.  They  indicated  that  they  would  prefer  to  have  the 
CTEIP  projects  managed  within  test  agencies,  rather  than  within  the  PMO.  They 
acknowledged  a  need  to  get  out  to  the  PMOs  more  often,  but  were  pressed  to  find 
personnel  to  accomplish  this.  Most  of  those  interviewed  worked  in  severely  understaffed 
offices  with  many  requirements  on  their  resources.  There  was  a  mixed  response  on  the 
degree  to  which  they  should  reach  out  toward  the  PMO,  some  feeling  the  avenues  were 
already  in  place  for  the  PMO  to  get  ah  the  T&E  support  necessary.  But  ah  agreed  that 
there  was  room  for  the  OSD  and  Headquarters  level  management  to  reach  out  to  inform 
and  educate  the  PMOs  on  the  value  of  CTEIP  and  the  total  T&D  Investment  process. 
There  were  no  current  PMO  sponsored  or  managed  CTEIP  projects,  although  they  were 
working  very  closely  with  several  PMOs,  even  to  the  extent  of  having  a  Test  Officers  co¬ 
located  in  Headquarters  offices  while  test  requirements  were  being  defined.  It  is  at  this 
stage  or  earlier  that  the  CTEIP  management  interviewees  felt  the  PMO  should  initiate 
dialog  to  define  their  T&E  requirements  and  test  shortfalls. 
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IV.  DATA  ANALYSIS 


A.  INTRODUCTION 

This  chapter  presents  an  analysis  of  the  data  that  was  presented  in  the  previous 
chapter.  The  subject  of  the  analysis  was  the  data  obtained  to  answer  the  primary  thesis 
questions:  “How  can  a  PMO  insert  themselves  into  the  CTEIP  Needs  and  Solutions 
process  in  order  to  obtain  funding  for  test  program  developments?”;  “Would  funding 
from  CTEIP  be  of  value  to  the  PMO’s  test  program?”;  and  “How  can  a  PMO  become 
aware  of  the  availability  and  applicability  of  CTEIP  funding  for  their  test  programs?” 

B.  AREAS  OF  ANALYSIS 

After  analyzing  the  literature  research  and  the  interview  responses,  it  was  decided 
that  the  analysis  would  concentrate  on  key  factors  within  the  PMO  and  the  CTEIP 
management  and  oversight  agencies.  These  key  factors  were  detennined  to  be  common 
areas  contributing  to  the  current  disconnect  between  the  CTEIP  management  and  the 
PMOs.  They  were  serving  as  barriers  to  the  PMO  insertion  into  the  T&E  investment 
process.  The  areas  are  identified  in  the  following  sections. 

1.  Organizational  Culture 

The  culture  within  an  organization  is  defined  by  the  norms  and  traditions  that 
make  up  that  organization.  These  norms  and  traditions  also  define  how  an  organization 
reacts  to  internal  and  external  influences.  Organizational  culture  can  be  defined  as  a 
framework  of  attitudes,  values,  behavioral  norms,  and  expectations  that  are  shared  by  the 
members  of  an  organization.  [Ref.  9:p.  256]  An  analysis  of  the  organizational  culture  of 
the  PMOs  and  the  CTEIP  management  structure  provides  insights  pertinent  to  this  thesis. 

2.  Communication 

Communication  is  the  process  by  which  information  is  sent  out  and  information  is 
received.  [Ref.  9:  p.  125]  The  communication  processes  and  policies  are  a  result  of  the 
culture  that  is  evident  within  an  organization.  But  looking  at  this  on  its  own  also 
examines  the  extent  of  the  effort  made  by  the  PMOs  and  CTEIP  to  react  within  their 
overlapping  communication  environments. 
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3.  Skill  Development 

Organizations  consist  of  personnel.  The  knowledge  base  of  these  employees  is  a 
key  contributor  to  the  effectiveness  of  those  organizations.  Classroom  training  is  but  one 
method  of  expanding  a  knowledge  base.  The  analysis  of  the  data  includes  a  review  of  the 
knowledge  that  PMOs  had  of  CTEIP  and  the  identification  of  skills  necessary  within  both 
organizations  to  facilitate  interaction. 

4.  Program  Policy  and  Strategy 

The  internal  policies  and  strategies  adopted  by  organizations  also  have  an  impact 
on  the  outcomes  obtained.  It  is  in  this  area  that  the  data  pertaining  to  the  Test  Investment 
process,  CTEIP  planning  process,  and  CTEIP  sub-project  selection  criteria  was  analyzed. 
A  review  of  these  policies  and  strategies  within  the  context  of  the  PMOs  and  CTEIP 
interaction  provided  some  significant  insights. 

C.  ANALYSIS 

1.  U.  S.  Army  Program  Management  Offices,  T&E  Divisions 
a.  Organizational  Culture 

The  PMO  organizational  culture  is  a  very  success-oriented  environment. 
The  personnel  generally  have  many  years  experience  in  other  PMOs  and  bring  to  the 
office  an  understanding  of  the  office  structure  and  work  ethic  required  to  handle  the  task 
of  managing  the  development  of  a  DoD  weapon  system.  Internally,  there  is  much 
competition  between  the  varying  functional  divisions,  but  that  is  usually  handled  in  good 
fun.  The  personnel  are  nothing  but  professional,  and  in  the  better  situations,  form  a  well- 
bonded  team,  each  managing  a  portion  of  the  whole.  Externally,  the  PMO  usually  puts 
forth  a  united  front.  Many  of  the  decisions  made  within  a  PMO  have  a  basis  in  politics 
rather  than  actual  program  performance.  Due  to  this,  the  PMO  personnel  are  usually 
reticent  to  put  full  trust  in  other  agencies  when  the  cost  or  schedule  of  the  program  is 
dependent  on  the  agencies’  actions.  This  is  especially  true  when  it  comes  to  any  OSD- 
level  office  or  test  agencies.  These  agencies  have  final  decision  authority  on  many 
aspects  within  the  PMO.  The  PMO  would  rather  maintain  a  close-hold  attitude  on 
potential  sensitive  information,  as  in  unavailable  test  resources,  than  to  admit  a  potential 
program  deficiency  to  someone  from  OSD  or  the  test  community.  Test  and  Integration 
IPTs  allow  the  PMO  to  maintain  a  certain  level  of  control  and  still  give  the  appearance  of 
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coordination.  The  PMO  is  also  reluctant  to  tie  the  success  of  their  test  program  to  a 
particular  technology  or  development  that  may  not  be  available  to  meet  the  program 
schedule.  All  these  contribute  to  the  extent  that  a  PMO  would  attempt  to  participate  in 
the  T&E  Investment  process. 

b.  Communications 

The  PMO  is  very  good  at  communicating  with  the  agencies  that  contribute 
what  they  perceive  as  value  to  their  program,  but  this  communication  falls  within  the 
areas  that  the  PMO  wishes  to  control.  By  being  proactive  in  communicating  with  internal 
divisions  and  external  agencies,  the  PMO  tries  to  head-off  any  attempts  to  negatively 
impact  the  program.  This  means  that  the  external  communications  are  usually  very 
controlled  to  maintain  a  distance  between  the  PMO  and  potentially  adversarial  agencies 
such  as  OSD  and  the  Test  communities.  The  test  community  has  perceived  this  as  the 
PMOs  not  planning  well  for  future  test  needs,  and  as  not  being  very  forthcoming  in 
expressing  the  true  test  requirements.  More  open  communications  between  these  groups 
could  begin  to  change  these  perceptions.  The  active  use  of  true  IPTs  is  beginning  to 
make  a  difference. 

c.  Skills  Development 

The  personnel  interviewed  for  this  research  were  all  senior  team  leads  and 
division  chiefs  within  the  test  area  of  the  PMO,  occupying  positions  that  are  classified  as 
Acquisition  Critical  positions.  This  requires  undergoing  a  certification  process  based  on 
training  in  the  required  field  or  work  experience  of  a  defined  length  of  time.  Once  a 
Level  III  certification  is  achieved,  there  is  a  continuing  educational  requirement  of  eighty 
hours  of  training  every  two  years.  Since  a  sufficiently  high  number  of  the  interviewees 
were  unaware  of  CTEIP  and  of  how  it  could  benefit  their  program,  the  certification 
training  obtained  was  obviously  lacking.  Most  had  attended  conferences  as  part  of  the 
continuing  education,  but  these  predominately  were  in  technology  areas  relating  to  their 
program  or  to  service  and  program  level  conferences.  Very  few  had  attended  T&E 
specific  conferences  on  a  regular  basis. 

d.  Program  Policy  and  Strategy 

PMOs  are  very  IPT  and  team  oriented.  It  is  usually  necessary  to  get  input 
from  many  agencies  in  order  to  get  through  milestone  reviews.  To  this  end,  PMOs  have 
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always  employed  some  form  of  IPT  structure.  But  the  acquisition  strategy  of  the 
program  is  the  ruling  document.  While  there  are  other  documents  that  dictate  action  at 
different  phases,  the  PMOs  do  all  things  possible  to  keep  the  schedule  and  cost  of  the 
program  intact.  In  the  cases  of  the  inevitable  schedule  slips  and  cost  overruns,  the  test 
program  suffers  from  budget  reductions  and  schedule  crunches.  This  does  not  build 
goodwill  between  the  PMOs  and  the  test  community. 

2.  CTEIP  Management 

a.  Organizational  Culture 

The  test  community  is  very  parochial  in  nature.  The  test  agencies, 
activities,  and  ranges  function  as  closed  environments  in  much  the  same  way  a  PMO 
functions.  There  is  doubt  by  the  test  community  that  the  PMOs  will  actually  allow  the 
test  agencies  to  conduct  valid  tests  on  their  systems.  They  construe  the  constant  shifting 
of  test  requirements  within  a  PMO  as  lack  of  commitment  rather  than  a  response  to 
political  pressure. 

b.  Communications 

The  CTEIP  management  and  oversight  agencies  have  a  very  complex 
network  of  communications  and  coordinations  that  are  required  to  support  the  T&E 
Investment  process.  Yet  this  elaborate  network  lacks  a  fundamental  participant,  the 
PMO.  The  program  execution  guides  that  define  the  nature  of  CTEIP  and  the  entire 
project  selection  process  are  available  to  all  who  wish  to  use  them,  but  they  are  primarily 
circulated  among  the  test  community.  Internet  access  to  information  and  points  of 
contact  exist,  but  are  outdated  and  incorrect.  Portions  of  the  T&E  community  Internet 
access  require  approved  access,  if  the  PMO  knows  who  to  contact  for  that  access.  The 
CTEIP  management  offices  suffer  the  same  time  and  personnel  resource  constraints  as 
the  PMOs.  Therefore,  one-on-one  communication  with  each  and  every  PMO  is  not 
feasible. 

c.  Skills  Development 

The  personnel  within  the  test  agencies  are  very  well  versed  in  the  activities 
and  requirements  within  the  test  community.  The  Army  T&E  community  is  part  of  the 
Acquisition  Corps  and  has  the  same  certification  and  continuing  education  requirements 
as  the  PMO  personnel.  The  test  community,  though  trained,  lacks  the  fundamental 
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understanding  of  PMOs.  This  makes  understanding  the  needs  of  a  PMO  hard  to 
accomplish. 

d.  Program  Policy  and  Strategy 

The  policies  and  strategies  espoused  within  the  Test  Investment  process  do 
not  include  input  from  the  PMOs.  While  there  could  be  a  very  indirect  inclusion  of  the 
PMOs,  with  the  assumption  that  the  test  agencies  receive  their  requirements  for  Needs 
and  Solutions  from  the  PMOs,  this  is  not  always  the  case,  at  least  not  within  the  Army 
PMOs  surveyed.  It  is  probable  that  the  PMO  TEMPs  are  being  reviewed  for  the  test 
technologies  and  resources  required  to  test  the  system.  PMOs  are  not  going  to  submit 
any  official  document  that  has  requirements  they  or  someone  else  may  not  meet. 
Therefore,  the  majority  of  the  T&E  Needs  and  Solutions  come  from  within  the  test 
community.  This  is  an  aspect  of  the  parochial  nature  of  the  test  community,  which 
fosters  mistrust  from  the  PMOs. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

The  intent  of  this  thesis  was  to  review  CTEIP  and  evaluate  the  extent  to  which 
PMOs  are  involved  in  the  submission  of  test  and  evaluation  needs  from  which  CTEIP 
projects  are  selected.  The  objective  was  to  determine  if  the  CTEIP  managers  for 
potential  projects  directly  or  indirectly  solicit  PMOs,  and  how  the  PMOs  can  more 
directly  insert  themselves  into  the  project  selection  process.  The  research  revealed  that 
both  CTEIP  Managers  and  PMOs  felt  there  was  definite  value  to  the  PMOs  being 
involved  in  the  TE&  investment  process;  however,  they  disagreed  as  to  how  the  PMO 
should  be  involved.  A  detailed  analysis  of  the  current  T&E  investment  process,  the 
CTEIP  planning  process,  and  the  CTEIP  sub-project  selection  process  was  conducted  and 
nowhere  was  there  mention  of  PMO  direct  or  indirect  involvement  in  the  processes.  This 
chapter  presents  some  conclusions  drawn  from  the  analysis  of  the  research  data  as  well  as 
answers  to  the  research  questions.  Recommendations  for  the  PMOs  and  the  CTEIP 
management  are  also  provided.  Finally,  suggested  future  research  areas  are  identified 
that  could  expand  on  the  ideas  presented  in  this  thesis. 

B.  CONCLUSIONS 

1.  U.  S,  Army  Program  Management  Offices 

The  prevalent  culture  in  the  PMOs  promotes  planning  success-oriented  programs. 
While  this  may  be  positive  in  nature,  it  can  adversely  influence  the  choices  made  in  the 
planning  process.  Unavailable  test  items  will  not  always  be  identified  as  a  necessary 
requirement  due  to  the  mistrust  that  it  will  adversely  affect  the  PMO’s  test  program,  and 
in  turn,  the  total  program.  There  is  especially  a  level  of  mistrust  toward  OSD  agencies. 
A  statement  of  a  test  resource  deficiency  could  end  up  being  interpreted  that  the  program 
is  un-executable.  Therefore,  the  PMOs  paint  a  cautiously  rosy  picture,  hoping  that  the 
necessary  technology  comes  along  to  test  their  systems,  but  reticent  to  jump  on  the 
bandwagon  to  support  the  test  asset  development.  This  is  detrimental  to  both  the  PMO’s 
success  and  the  T&E  community’s  success. 

The  PMO  T&E  workforce  is  not  as  informed  as  it  should  be  to  adequately  plan 
and  implement  successful  test  programs.  Despite  Acquisition  Level  III  certification  in 
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the  T&E  career  field,  they  lack  a  fundamental  understanding  of  the  T&E  Investment 
process  and  the  T&E  agencies  and  organizations  that  exist  to  aid  the  PMO  in  test 
planning  and  resource  allocation.  They  lack  a  working  knowledge  of  similar 
technologies  being  tested  within  the  Army  and  within  other  services.  This  seems  to  stem 
from  the  prevalent  PMO  culture,  which  assumes  that  the  T&E  community  is  only  there  as 
support  and  is  not  aware  of  the  PMO-specific  issues.  “You  don’t  understand  my  program 
strategy”  is  an  often-used  phrase  at  times  when  coordination  is  attempted.  The  PMO  is 
also  reluctant  to  approve  attendance  and  participation  by  its  personnel  at  meetings  and 
conferences  not  directly  related  to  the  program.  This  isolates  the  PMO  professionals 
from  a  valuable  resource  of  ideas. 

The  PMO  program  strategy  and  planning  also  takes  on  an  “us  against  them” 
attitude.  While  cooperation  and  IPTs  are  actively  promoted,  the  PMO  stands  staunchly 
by  its  program  planning,  and  comments  or  ideas  by  all  outsiders  are  perceived  as  being 
detrimental  to  the  program.  The  words  of  cooperation  and  coordination  with  the 
strategies  are  not  implemented  to  the  full  benefit  of  the  program. 

2.  CTEIP  Management 

The  CTEIP  management  and  T&E  oversight  agencies  have  their  own  prevailing 
culture.  They  understand  the  importance  T&E  plays  within  the  development  of  a  system, 
yet  they  feel  that  the  PMOs  do  not  understand.  PMOs  are  forever  cutting  budgets  and 
schedules,  and  creating  scenarios  to  save  time  and  money.  Therefore,  they  could  not 
possibly  understand  the  importance  of  testing  and  would  not  have  any  valid  input  to  the 
T&E  investment  process.  While  that  is  not  always  the  case,  it  seems  to  be  a  common 
perception. 

The  T&E  professionals  that  ultimately  manage  the  development  of  investment 

projects  are  not  as  well  versed  in  the  acquisition  management  principles  as  they  should 

be.  Some  of  them  are  part  of  the  Acquisition  Workforce  and  Acquisition  Corps, 

especially  in  the  Army  T&E  community,  but  very  few  have  ever  been  part  of  a  true 

acquisition  office,  such  as  a  PMO.  The  acquisition  training  is  very  good,  but  it  does  not 

replace  the  experience  gained  working  in  a  PMO.  Also,  the  times  that  they  do  attend  the 

program  reviews,  their  main  interest  is  in  the  technical  briefings,  not  the  programmatic 

briefings.  This  is  understandable,  but  raises  the  question  of  why  a  test  agency  would  not 
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want  to  be  aware  of  any  programmatic  issues  that  might  ultimately  affect  the  test 
program?  Again,  coordination  and  cooperation  between  the  PMO  and  the  test 
community  is  sorely  needed. 

Generally,  the  CTEIP  management  acknowledges  the  CTEIP  policies  and 
strategies  that  involve  coordinating  with  PMOs.  Unfortunately,  they  do  not  have  the 
manpower  resources  necessary  to  individually  provide  information  briefings  to  each 
PMO.  They  do  recognize  that  the  success  of  the  CTEIP  projects  depends  on  the  support 
from  and  championship  by  major  system  development  program  offices.  Yet,  in  most 
cases,  they  rely  on  second  or  third  hand  references  for  this  support  or  from  a  top-down 
edict  without  coordinating  directly  with  the  PMO. 

C.  RESEARCH  QUESTIONS 

1.  How  Can  a  Program  Management  Office  (PMO)  Become  Aware  of 
the  Availability  and  Applicability  of  CTEIP  Funding  for  Their  Test 
Programs? 

The  first  way  a  PMO  can  become  aware  of  CTEIP,  if  it  is  applicable  to  their 
program,  is  to  talk  early  and  often  with  the  T&E  community.  Start  with  the  T&E  points 
of  contact  at  their  own  command.  Then,  coordinate  with  the  local  TEMAC  on  the  timing 
of  the  Needs  and  Solutions  data  call  and  possible  technology  areas  that  could  become 
CTEIP  proposals.  They  should  work  with  the  ATEC  representative  to  the  PMO’s  Test 
and  Integration  Working  Group  or  T&E  Integrated  Product  Team  (IPT)  to  develop  a 
strategy  for  obtaining  the  needed  test  assets,  instrumentation,  and  improved  infrastructure 
necessary  to  test  their  program.  Finally,  they  should  look  to  the  OSD  CTEIP 
management  for  coordination  with  other  service  programs  that  might  have  similar 
technology  needs. 

However,  even  before  the  coordination  takes  place,  the  PMO  test  personnel  must 
be  knowledgeable  on  the  total  test  investment  process.  They  must  go  beyond  the 
Acquisition  Test  and  Evaluation  Level  III  Certification  training  and  seek  out  the 
information  through  talking  with  the  T&E  community,  attending  conferences  that  address 
the  T&E  Investment  planning,  and  look  forward  and  outward  beyond  the  stovepipe  of 
their  own  program.  This  will  involve  some  changes  in  culture  within  the  PMO  to  reach 
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out  and  trust  the  ATEC  and  OSD  level  test  community,  but  effective  training,  education, 
and  experience  will  be  the  beginnings  of  that  culture  change. 

2.  Would  Funding  from  CTEIP  be  of  Value  to  the  PMO’s  Test 
Program? 

Anything  that  will  assist  the  Program  Manager  to  field  a  better  product  in  a 
shortened  timeframe  for  less  money  is  of  value.  Test  programs  for  any  acquisition 
system  make  up  a  large  portion  of  the  overall  development  cost.  The  funding  value  can 
result  from  a  directly  sponsored  CTEIP  project  managed  by  the  PMO  to  address  short¬ 
term,  previously  un-addressed  test  requirements.  The  value  can  come  from  teaming  with 
another  service  or  a  test  agency  to  address  a  multi-service  need  that  will  aid  in  the  testing 
of  a  new  technology  for  years  in  the  future.  The  value  can  result  from  the  PMO  clearly 
identifying  their  future -technology  areas,  adding  their  weight  to  CTEIP  projects 
development,  and  improving  the  overall  test  infrastructure.  All  of  these  possibilities  add 
value  to  the  PMO  test  program  by  providing  resources  of  time,  money,  and  equipment  or 
facilities  to  an  already  stretched-thin  program.  All  of  these  will  provide  the  PMO  with  a 
better  test,  and  in  turn,  a  better  weapon  system  will  be  delivered  to  the  field  and  soldier. 
This  is  the  value  to  the  PMO. 

3.  How  Can  a  PMO  Insert  Themselves  into  the  CTEIP  Needs  and 
Solution  Process  in  Order  to  Obtain  Funding  for  Test  Program 
Developments? 

To  become  an  integral  part  of  the  T&E  Investment  process,  the  PMO  must  first 
educate  its  personnel  on  the  process;  become  familiar  with  the  process  and  the  key 
agencies  that  make  decisions  throughout  the  process;  and  find  the  area  or  part  of  the 
process  where  they  can  contribute  the  most  input.  Next,  the  PMO  should  coordinate 
constantly  with  the  test  agencies  that  are  already  a  part  of  the  Needs  and  Solutions 
process.  Even  if  direct  involvement  is  not  the  best  way  to  proceed,  coordinating  with  the 
test  agencies  on  the  test  needs  of  particular  programs  and  technologies  will  eventually 
pay  dividends  through  improved  test  capabilities  at  the  test  agencies.  Finally,  the  PMO 
test  personnel  should  become  familiar  with  the  process  for  submitting  Needs  and 
Solutions,  and  start  participating  in  the  process.  It  may  take  some  time,  which  is  at  a 
premium  in  a  PMO,  but  eventually  the  participation  will  pay  off. 
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4.  What  are  Some  Strategies  that  the  CTEIP  Management  Could  Use  to 
Educate  and  Offer  Assistance  to  PMO  Personnel? 

The  CTEIP  management  could  start  with  some  simple  things  to  educate  and  assist 
PMOs.  The  easiest  way  is  to  ensure  the  CTEIP  and  DOT&E  websites  are  up-to-date  and 
accurate.  This  can  become  a  gateway  for  the  PMO  to  enter  the  CTEIP  and  T&E 
Investment  process.  However,  incorrect  and  out  of  date  information  can  stop  the  PMO  in 
its  tracks  and  continue  the  disconnect  that  exists  today.  CTEIP  and  the  Army  T&E 
agencies  already  participate  and  sponsor  conferences  and  seminars  that  promote  T&E 
awareness.  Finding  a  way  to  advertise  these  to  the  PMO  test  personnel  and  encouraging 
their  involvement  at  the  conferences  would  spread  some  of  the  T&E  awareness  into  the 
PMOs. 

D.  RECOMMENDATIONS 

1.  U.  S.  Army  Program  Management  Offices 

The  PMOs  should  coordinate  more  actively  with  the  available  test  organizations 
and  agencies.  The  IPT  process  has  improved  communication  and  coordination 
somewhat,  but  there  is  still  hesitancy  and  mistrust  in  presenting  potential  program 
weaknesses  to  the  T&E  agencies  and  especially  to  anyone  in  OSD.  It  is  felt  that  if  an 
unavailable  test  asset  or  capability  is  identified  as  a  definite  requirement,  then  several 
things  might  happen.  The  program  could  be  deemed  un-executable;  the  program  could 
be  made  to  pay  for  any  needed  upgrades  or  developments  to  obtain  the  test  capability;  or 
the  program  could  suffer  schedule  slippages  that  would  affect  the  overall  program  cost. 
This  is  a  cultural  manifestation  within  the  PMO  organizations.  This  must  be  overcome. 
There  needs  to  be  dialog  early  in  the  program  life  cycle  between  the  PMO  and  the  T&E 
community  to  plan  out  all  needed  T&E  investments  that  are  required  to  adequately  test 
the  planned  development  system.  These  discussions  should  not  tie  the  hands  of  the  PMO 
and  their  program  execution,  but  the  PMO  must  be  willing  to  actively  support  the  T&E 
development  and  not  be  ambiguous  about  the  capability  need. 

The  PMOs  should  recognize  that  personnel  have  a  variety  of  training  experiences 
available  to  them.  Level  III  acquisition  certification  is  not  the  end  of  the  learning  process 
within  a  given  career  field.  Involvement  in  T&E  specific  conferences  and  symposiums 
allows  exposure  to  a  variety  of  technologies,  capabilities,  and  contacts  in  the  T&E 
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community.  This  serendipitous  learning  can  reap  endless  value,  especially  for  start-up 
PMOs  and  those  seeking  to  integrate  new  technologies  into  their  systems.  The 
opportunity  to  attend  as  many  T&E  related  events  as  possible  should  be  a  priority  of  the 
continuous  learning  experience.  Likewise,  attendance  at  program  reviews  of  like- 
systems,  even  in  the  other  services,  could  provide  even  more  benefit  to  the  PMO.  T&E 
knowledge  and  resources  must  be  learned  so  that  they  can  be  utilized  in  the  most  efficient 
and  expedient  manner. 

Finally,  the  PMO  program  policies  and  program  strategies  must  support  the  T&E 
investment  process.  Testing  is  a  vital  piece  of  the  development  process.  Selling  it  short 
by  reducing  the  number  of  tests  and  reducing  the  test  budget  can  be  a  reality.  But  every 
effort  must  be  made  to  maintain  the  integrity  of  the  system  testing  by  ensuring  enough 
time  and  resources.  We  owe  it  to  the  soldiers  that  will  be  using  our  systems. 

2.  CTEIP  Management 

The  CTEIP  management  and  oversight  agencies,  as  well  as  the  T&E  Investment 
planning  community,  should  include  the  PMOs  at  every  possible  step.  While  they  may 
not  be  directly  tasked  to  respond  to  the  Needs  and  Solutions  data  calls,  the  PMOs  should 
be  notified  when  the  T&E  agencies  have  received  the  data  call  so  that  investment  needs 
or  possible  solutions  can  be  submitted  through  the  appropriate  channels.  The  PMOs  have 
a  direct  contractual  link  with  industry.  This  link  could  provide  knowledge  of  on-going 
and  state-of-the-art  technologies  that  are  being  planned  and  developed  for  current 
acquisition  developments.  The  PMOs  are  the  ultimate  customers  of  the  T&E  community 
and  they  should  be  included  in  the  identification  of  pressing  T&E  issues  and  needs.  This 
would  be  a  mutually  beneficial  partnership  for  both  the  PMO  and  the  CTEIP  and  T&E 
Investment  process. 

The  T&E  Community  sponsors  many  conferences  and  symposiums  for  its 
membership.  A  more  concerted  effort  to  include  PMO  personnel  on  the  distribution  list 
for  these  events  could  provide  valuable  opportunities  for  both  organizations.  One  way  to 
involve  PMOs  would  be  to  invite  some  of  the  PMO  T&E  leads  to  the  CTEIP  Mid-Year 
Review  Briefings.  This  would  be  an  excellent  way  to  become  familiar  with  the  types  of 
investment  programs  that  are  sponsored  and  the  type  of  work  done  across  the  services. 

Another  such  conference  that  has  been  held  in  Huntsville,  AL  since  1998  is  the  Army 
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T&E  Days  Conference  and  Exhibition.  This  event  is  sponsored  by  TEMA  and  the 
AMCOM  TEMAC  and  involves  3-4  days  of  speakers,  working  sessions,  and  exhibits  that 
showcase  the  best  that  Anny  T&E  has  to  offer.  Coupled  with  this  event  is  a  day  of  T&E 
specific  training  and  special  topic  courses.  It  is  possible  that  either  as  part  of  the 
briefings  or  as  part  of  the  training  sessions,  more  in-depth  infonnation  on  the  T&E 
Investment  process  and  CTEIP  could  be  shared  with  the  attendees.  CTEIP  is  a  success 
story  and  should  be  shared.  The  educational  process  could  also  be  furthered  by  clearly 
identifying  in  the  exhibition  the  programs  that  received  CTEIP  sponsorship. 

Finally,  the  T&E  investment  policy  and  strategy  should  explicitly  address  active 
coordination  and  cooperation  with  the  PMOs.  While  this  is  currently  in  the  policy  and 
strategy  plans,  the  implementation  of  this  has  fallen  to  the  wayside.  The  culture  within 
the  T&E  community  needs  to  begin  to  foster  a  cooperative  attitude  in  which  resources  are 
shared.  The  end  result  of  both  the  T&E  community  and  the  PMOs  efforts  is  the  best 
system  possible  for  the  warfighter. 

E.  FUTURE  RESEARCH  AREAS 

During  the  course  of  this  research  and  analysis,  several  areas  were  highlighted  as 
topics  for  potential  research  papers,  thesis  topics,  and  long-tenn  projects: 

1.  Organizational  Systems  Analysis 

Propose  that  the  interaction  between  the  PMO  and  the  Test  Investment  Process  be 
researched  and  analyzed  utilizing  the  Systems  Organizational  Framework.  This  could 
provide  valuable  insight  into  the  inter-relationships  of  the  two  separate  organizations  and 
provide  guidance  on  how  they  could  interrelate  better. 

2.  Test  Culture 

The  culture  prevalent  in  the  test  community  impacts  the  interaction  this 
community  has  with  all  other  communities,  especially  the  acquisition  community. 
Further  research  could  dissect  that  culture  to  discover  key  factors  impacting  the  culture 
and  impacting  the  interaction  with  other  cultures. 

3.  PMO/Acquisition  Culture 

The  culture  of  the  PMO  is  very  different  from  that  of  the  test  community.  A 
comparison  of  the  two  cultures  could  be  done.  The  PMO  has  a  much  more  political 
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outlook  that  impacts  the  decisions  made  within  the  program  strategy.  The  effects  of  this 
culture  on  successful  or  unsuccessful  test  program  could  be  a  valid  research  topic. 

4.  T&E  Strategic  Planning  and  Management 

The  total  test  investment  process  could  benefit  from  the  development  of  a 
strategic  management  plan.  Increased  interest  and  funding  on  the  need  for  advances  and 
improvements  in  the  test  infrastructure  has  brought  the  focus  to  looking  to  industry  and 
the  technological  advances  that  are  being  developed  both  for  weapon  systems  and  for  test 
instrumentation.  An  analysis  of  selected  areas  or  programs  could  identify  the  appropriate 
model  or  models  to  be  used  in  the  development  of  the  plan.  Once  a  plan  is  in  place,  then 
the  priorities  could  focus  on  the  actual  implementation  of  the  plan.  This  same  approach 
could  be  accomplished  from  the  PMO  standpoint  toward  a  total  strategic  management 
plan  for  a  test  program. 
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AACM-FWG 

AoA 

AMC 

AMCOM 

AMSAA 

ARL 

ATEC 

AVCSA 

BMDO 

BoD 

BoD  (ES) 
BoD  (ESS) 
BoOD 


C4I 

CECOM 

CTEIP 

DA 

DDDR&E 

DDR&R 

DIA 

DoD 

DOT&E 

DT&E 


APPENDIX.  LIST  OF  ACRONYMS 

Army  Acquisition  Career  Management  -  Functional 

Working  Group 

Analysis  of  Alternatives 

United  States  Army  Materiel  Command 

United  States  Army  Aviation  and  Missile  Command 

U.  S.  Army  Materiel  Systems  Analysis  Activity 

United  States  Anny  Research  Laboratory 

United  States  Army  Test  and  Evaluation  Command 

Assistant  Vice  Chief  of  Staff  of  the  Army 

Ballistic  Missile  Defense  Organization  (see  MDA) 

Board  of  Directors 

Board  of  Directors  (Executive  Secretariat) 

Board  of  Directors  (Executive  Secretariat  Staff) 

Board  of  Operating  Directors 

Command,  Control,  Communications,  and  Intelligence 
Command,  Control,  Communications,  Computers  and 
Intelligence 

U.  S.  Army  Communications  and  Electronics  Command 
Central  Test  and  Evaluation  Investment  Program 

Department  of  the  Army 

Deputy  Director,  Defense  Research  and  Engineering 
Deputy  Director,  Resources  and  Ranges 
U.  S.  Defense  Intelligence  Agency 
United  States  Department  of  Defense 
Director  of  Operational  Test  and  Evaluation 
Director,  Test  and  Evaluation 
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DT&E 
DTC 
DTTSG 
DUSA  (OR) 


EA 

ES 

FOR 

FMS 

FY 

GM 

HQ 

IOC 

IPT 

JIM 

JITC 

JPO 

LCSP 

LFT&E 

LOI 

MDA 

MEDCOM 

MOA 


Developmental  Test  and  Evaluation 

Developmental  Test  Command 

Defense  Test  and  Training  Steering  Group 

Deputy  Under  Secretary  of  the  Army  for  Operations 

Research 

Executive  Agents 
Executive  Secretariat 

Functional  Chief  Representative 
Foreign  Materiel  Sales 
Fiscal  Year 

General  Management 

Headquarters 

U.  S.  Army  Industrial  Operations  Command 
Integrated  Product  Team 

Joint  Improvement  and  Modernization 
Joint  Interoperability  Test  Command 
Joint  Program  Office 

Life  Cycle  Support  Plan 
Live  Fire  Test  and  Evaluation 
Letter  of  Instruction 

Missile  Defense  Agency  (formerly  BMDO) 

U.  S.  Army  Medical  Command 
Memorandum  of  Agreement 
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MRTFB 

OASARDA 

OCOE 

ODCSINT 

ODCSLOG 

ODCSOPS 

ODICS4 

O&M 

OSD 

OT 

OTA 

OT&E 

OTICC 

OTSG 

OUSD  (A) 

OUSD  (A&T) 


PD 

PE 

PEM 

PEO 

PM 

PM,  ITTS 
PMO 
PMP 
POM 


Major  Range  and  Test  Facility  Base 

Office  of  the  Assistant  Secretary  of  the  Anny,  Research 

Development  and  Acquisition 

Office  of  the  Chief  of  Engineers 

Office  of  the  Deputy  Chief  of  Staff  for  Intelligence 

Office  of  the  Deputy  Chief  of  Staff  for  Logistics 

Office  of  the  Deputy  Chief  of  Staff  for  Operations  and 

Plans 

Office  of  the  Director  for  Information  Systems  for 

Command,  Control,  Communications,  and  Computers 

Operations  and  Maintenance 

Office  of  the  Secretary  of  Defense 

Operational  Test 

Operational  Test  Agency 

Operational  Test  and  Evaluation 

OSD  Test  Investment  Coordinating  Committee 

Office  of  the  Surgeon  General 

Office  of  the  Under  Secretary  of  Defense  (Acquisition) 
Office  of  the  Under  Secretary  of  Defense  (Acquisition  and 
Technology) 

Project  Director 
Program  Element 
Program  Element  Manager 
Program  Executive  Office 
Program  Manager 

PM,  Instrumentation,  Targets,  and  Threat  Simulators 

Program  Management  Office 

Program  Master  Plan 

Program  Objective  Memorandum 
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PPBS 

REP 

REP  WG 

SBCCOM 
Sec  Def 
SMDC 
SMP 

ssc 

STRICOM 


T&E 

TAACOM 

TCMP 

TCRD 

TECOM 

TEMA 

TEMAC 

TEMP 

TERC 

TERIB 

URIC 

TIS 

TPD 

TRADOC 

TRAG 

TRMP 

TTD&D 


Planning,  Programming,  and  Budgeting  System 

Resource  Enhancement  Project 

Resource  Enhancement  Project  Working  Group 

U.  S.  Army  Soldier  and  Biological  Chemical  Command 
Secretary  of  Defense 

U.  S.  Strategic  Missile  Defense  Command 

Subproject  Management  Plan 

U.  S.  Army  Soldier  Support  Command 

U.  S.  Army  Simulation,  Training,  and  Instrumentation 

Command 

Test  and  Evaluation 

U.  S.  Army  Tank- Automotive  and  Armament  Command 

Test  Capability  Master  Plan 

Test  Capability  Requirements  Document 

U.  S.  Army  Test  and  Evaluation  Command 

U.  S.  Army  Test  and  Evaluation  Management  Agency 

Test  and  Evaluation  Managers  Committee 

Test  and  Evaluation  Master  Plan 

Test  and  Evaluation  Resource  Committee 

Test  and  Evaluation  Reliance  Investment  Board 

Training  Instrumentation  Resource  Investment  Committee 

Test  Investment  Strategy 

Test  Package  Directive 

U.  S.  Army  Training  and  Doctrine  Command 

Test  Resource  Advisory  Group 

Test  Resource  Master  Plan 

Test  Technology  Development  and  Demonstration 
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u.  s. 

USD  (AT&L) 


United  States 

Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics 


VCSA  Vice  Chief  of  Staff  of  the  Army 

WG  Working  Group 


55 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


56 


LIST  OF  REFERENCES 


1.  United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and 
Evaluation  Investment  Program,  Joint  Improvement  and  Modernization  Projects 
Planning  and  Executive  Guide,  Washington,  D.C.:  Director,  Operational  Test  and 
Evaluation,  1  September  2000. 

2.  United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and 
Evaluation  Investment  Program,  Resource  Enhancement  Program  Planning  and 
Execution  Guide,  Washington,  D.C.:  Director,  Operational  Test  and  Evaluation, 
2000. 

3.  CTEIP  Program  Element  Manager,  “Central  Test  and  Evaluation  Investment 
Program  Briefing,”  Washington,  D.C.,  12  July  2001. 

4.  Hinton,  Derrick,  “Central  Test  and  Evaluation  Investment  Program,”  Washington, 
D.C.,  11  October  2000. 

5.  United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and 
Evaluation  Investment  Program,  Test  Technology  Development  and 
Demonstration  Project  Planning  and  Execution  Guide,  Washington,  D.C.: 
Director,  Operational  Test  and  Evaluation,  April  2000. 

6.  Test  and  Evaluation  Executive  Agent, 
[http://www.acq.osd.mil/te/pubdocs/agent.html],  9  November  2002. 

7.  United  States  Army  Test  and  Evaluation  Management  Agency, 
[http://www.hqda.anny.mil/tema/],  14  July  2001. 

8.  Test  and  Evaluation  Reliance  and  Investment  Process, 
[http://www.acq.osd.mil/te/pubdocs/reliance.html],  9  November  2002. 

9.  Greenberg,  Jerald,  Managing  Behavior  in  Organizations,  Prentice  Hall:  New 
Jersey,  1999. 


57 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


58 


BIBLIOGRAPHY 


“George  Washington  Chapter  Sponsors  Crosstalk  2002:  Transforming  New  Technologies 
into  T&E  Capabilities,”  The  ITEA  Journal  of  Test  and  Evaluation  23.3,  2002. 

Christie,  Thomas  P.,  “Interview,”  Interview  with  Robert  T.  Fuller  and  Edward  G.  Garza. 
The  ITEA  Journal  of  Test  and  Evaluation,  September-October  2002. 

CTEIP  Program  Element  Manager,  “Central  Test  and  Evaluation  Investment  Program 
Briefing,”  Washington,  D.C.,  12  July  2001. 

Director,  Operational  Test  and  Evaluation,  Operational  Test  and  Evaluation,  Resources 
and  Ranges,  CTEIP,  Director,  Operational  Test  and  Evaluation, 
[http://www.dote.osd.mil/rr/cteip.html],  14  April  2002. 

Greenberg,  Jerald,  Managing  Behavior  in  Organizations,  Prentice  Hall:  New  Jersey, 

1999.  ~  “  ~  ' 

Hatfield,  James,  Chief,  Technical  Engineering  Branch,  PM  Aircraft  Electronic  Systems, 
Personal  Interview,  3  September  2002. 

Hinton,  Derrick,  “Central  Test  and  Evaluation  Investment  Program,”  Washington,  D.C., 
1 1  October  2000. 

Hinton,  Derrick,  CTEIP  PEM,  DOT&E,  Resources  and  Ranges,  Telephone  Interview,  5 
November  2002. 

Joiner,  Michael,  Technology  Management  Division,  Developmental  Test  Command, 
Internet  Interview,  4  December  2002. 

Seglie,  Ernest  A.,  “The  Future  of  Test  and  Evaluation,”  The  ITEA  Journal  of  Test  and 
Evaluation  23.3,  2002. 

Strohl,  Suzanne,  Chief,  Instrumentation  Division,  ATEC;  DOT&E  REP  Working  Group 
Chairperson,  Internet  Interview,  22  November  2002. 

Test  and  Evaluation  Agent,  [http://www.acq.osd.mil/te/pubdocs/agent.html],  9  November 

2002. 

Test  Investment  Process,  [http://www.acq.osd.mil/te/pubdocs/reliance/html],  9  November 

2002. 

United  States  Army  Test  and  Evaluation  Management  Agency, 
[http://www.hqda.anny.mil/tema/],  8  October  2002  and  14  July  2001. 


59 


United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and  Evaluation 
Investment  Program  1998  Annual  Report,  Washington  D.C.:  Director,  Operational  Test 
and  Evaluation,  1998. 

United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and  Evaluation 
Investment  Program  2001  Annual  Report,  Washington,  D.C.:  Director,  Operational  Test 
and  Evaluation,  2002. 

United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and  Evaluation 
Investment  Program,  Joint  Improvement  and  Modernization  Projects  Planning  and 
Executive  Guide,  Washington,  D.C.:  Director,  Operational  Test  and  Evaluation,  2000. 

United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and  Evaluation 
Investment  Program,  Joint  Improvement  and  Modernization  Project  Planning  and 
Execution  Guide.  Washington,  D.C.:  Director,  Operational  Test  and  Evaluation,  1998. 

United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and  Evaluation 
Investment  Program,  Resource  Enhancement  Program  Planning  and  Execution  Guide, 
Washington,  D.C.:  Director,  Operational  Test  and  Evaluation,  2000. 

United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and  Evaluation 
Investment  Program,  Resource  Enhancement  Program  Planning  and  Execution  Guide, 
Washington  D.C.:  Director,  Operational  Test  and  Evaluation,  1998. 

United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and  Evaluation 
Investment  Program,  Test  Technology  Development  and  Demonstration  Project  Planning 
and  Execution  Guide,  Washington,  D.C.:  Director,  Operational  Test  and  Evaluation, 
2000. 

United  States,  Director,  Operational  Test  and  Evaluation,  Central  Test  and  Evaluation 
Investment  Program,  Test  Technology  Development  and  Demonstration  Project 
Procedures  Manual,  Washington  D.C.:  Director,  Operational  Test  and  Evaluation,  1998. 

United  States,  House  of  Representatives,  Conference  Report,  101st  Cong.,  1st  Sess.,  HR 
Rept.  101-345,  Washington  D.C.:  U.  S.  Government  Printing  Office,  1989. 

United  States,  House  of  Representatives,  Conference  Report,  101st  Cong.,  2nd  Sess.,  HR 
Rept.  101-938,  Washington  D.C.:  U.  S.  Government  Printing  Office,  1990. 

United  States,  House  of  Representatives,  DoD  Appropriations  Bill,  1990;  Report  of  the 
Committee  On  Appropriations,  101st  Cong.,  1st  Sess.,  HR  Rept.  101-208,  Washington 
D.C.:  U.  S.  Government  Printing  Office,  1989. 

United  States,  House  of  Representatives,  DoD  Appropriations  Bill,  1991;  Report  of  the 
Committee  On  Appropriations,  101st  Cong.,  2nd  Sess.,  HR  Rept.  101-822,  Washington 
D.C.:  U.  S.  Government  Printing  Office,  1990. 


60 


United  States,  Senate,  DoD  Appropriation  Bill,  1990;  Report  from  the  Committee  on 
Appropriations,  101st  Cong.,  1st  Sess.,  S  Rept.  101-132,  Washington,  D.C.:  U.  S. 
Government  Printing  Office,  1989. 

United  States,  Senate,  DoD  Appropriation  Bill,  1990;  Report  from  the  Committee  on 
Appropriations,  101st  Cong.,  2nd  Sess.,  S  Rept.  101-521,  Washington,  D.C.:  U.  S. 
Government  Printing  Office,  1990. 

Wagner,  Ray,  Director  of  Resources,  U.S.  Army  Test  and  Evaluation  Management 
Agency,  Telephone  Interview,  8  November  2002. 


61 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


62 


INITIAL  DISTRIBUTION  LIST 


1 .  Defense  Technical  Infonnation  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 

3.  Derrick  Hinton 

Director  of  Operational  Test  &Evaluation,  Resources  and  Ranges 
Washington,  D.C. 

4.  Raymond  Wagner 

U.  S.  Army  Test  and  Evaluation  Management  Agency 
Washington,  D.C. 

5.  Margaret  F.  Haack 

Ground-Based  Midcourse  Defense  Joint  Program  Office 
Huntsville,  Alabama 


63 


